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FOREWORD 

This  report  is  a  general  review  of  the  literature  pertaining  to  the  evaluation  of  quality  of  life 
programs  in  the  military  and  civilian  sectors.  Studies  are  cited  along  with  a  brief  discussion  of  the 
strengths  and/or  weaknesses  of  the  methodologies  employed  and  their  applicability  of  Navy  eval¬ 
uation  efforts. 

This  task  was  conducted  with  O&M.N  funding  and  is  one  of  two  reports  produced  under  Work 
Unit  9WRTT578.  The  report  is  intended  for  the  use  of  OP-15/NMPC-6E. 
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SUMMARY 


Problem 

To  develop  the  evaluation  capabilities  necessary  to  assure  the  effectiveness  of  Navy  quality  of 
life  programs  and  make  changes  and  improvements  where  indicated.  An  understanding  of  the  field 
of  evaluation  and  an  awareness  of  previous  evaluation  efforts  are  needed. 

Objective 

The  objective  of  this  report  is  to  summarize  the  results  of  a  literature  review  conducted  to 
identify  previous  efforts  to  evaluate  quality  of  life  programs  in  the  military  and  civilian  sectors. 

Background 

Evaluation  efforts  may  reflect  a  broad  range  of  experimental  rigor  from  the  one-shot,  post  hoc 
case  study  approach  in  which  a  single  program  is  examined  after  a  program  or  treatment  has  been 
given  to  fully  planned  approaches  based  on  the  principles  of  research  design.  Evaluation  studies 
may  also  include  an  assessment  of  the  cost-effectiveness  or  cost-benefit  of  the  program.  However, 
social  programs,  including  quality  of  life  programs,  are  difficult  to  evaluate  in  this  manner  due  to 
the  many  intangible  costs  and  benefits  often  associated  with  them. 

The  key  to  the  evaluation  process  is  measurement.  A  wide  variety  of  measures  can  be  used, 
which  generally  will  include  measures  of  performance,  effectiveness,  and  efficiency. 

Approach 

The  results  of  a  literature  review  conducted  to  identify  previous  efforts  to  evaluate  quality  of 
life  programs  were  summarized  by  program  area.  Within  each  area,  the  studies  found  were  cited 
along  with  a  brief  discussion  of  the  strengths  and/or  weaknesses  of  the  studies,  their  results,  and 
their  applicability  to  Navy  evaluation  efforts.  This  review  includes  military  program  evaluations 
found,  as  well  as  applicable  evaluations  from  civilian  programs. 

Conclusions 

Except  in  certain  areas,  there  is  a  pervasive  lack  of  research  designed  to  evaluate  quality  of  life 
programs.  The  majority  of  the  evaluation  efforts  uncovered  suffer  from  a  lack  of  control  and  the 
sole  reliance  on  subjective  data.  Subjective  evaluation  is  very  important  in  the  quality  of  life  field, 
but  should  not  be  used  at  the  exclusion  of  objective  measures  when  they  can  be  developed. 

Evaluations  become  dated  very  quickly  as  programs,  policies,  and  the  environment  change. 
Therefore,  the  validity  of  previous  evaluation  results  for  current  quality  of  life  program  is  dubious. 

In  general,  studies  evaluating  drug  and  alcohol  control  efforts,  health  promotion  programs,  and 
employer-sponsored  child  care  programs  were  more  rigorous  and  objective  than  those  in  other 
areas.  There  are  more  reports  of  measured  outcomes  linked  to  these  programs  and  many  of  the^e 
evaluations  assess  the  effects  of  such  programs  on  more  than  one  outcome. 


In  some  quality  of  life  arenas,  there  are  major  problems  evaluating  program  outcomes  because 
outcome  areas  are  not  clearly  defined  and  process  measures  are  not  collected.  The  problems  of 
separating  the  effects  of  one  program  (e.g.,  military  pay)  on  a  given  outcome  measure  (e.g., 
retention)  from  the  effects  that  all  other  concomitant  factors  in  the  environment  have  on  the  same 
outcome  measure  is  difficult  in  real-world  research. 

An  effective  approach  to  subsequent  evaluations  will  include  service  member  attitude  and 
satisfaction  assessment,  as  a  minimum;  objective  effectiveness  data,  whenever  obtainable;  and 
cost-effectiveness  analysis,  when  feasible. 

Recommendation 

It  is  recommended  that  efforts  to  evaluate  Navy  programs  should  include  attitude  and 
satisfaction  assessment  and  objective  effectiveness  data  as  a  minimum,  with  cost-effectiveness 
analysis  when  feasible. 
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INTRODUCTION 


Problem 

To  effectively  carry  out  its  quality  of  life  programs,  the  Department  of  the  Navy  must  evaluate 
their  effectiveness  and  make  changes  and  improvements  where  indicated.  However,  evaluating 
social  programs,  including  quality  of  life  programs,  is  difficult  because  many  of  the  outcomes  are 
intangible.  To  develop  an  effective  evaluation  program  in  this  area,  an  understanding  of  the  field 
of  evaluation  and  an  awareness  of  previous  evaluation  efforts  are  needed. 

Objective 

The  objective  of  this  report  is  to  summarize  the  results  of  a  literature  review  conducted  to 
identify  previous  effoits  to  evaluate  quality  of  life  programs  in  the  military  and  civilian  sectors. 

Background 

In  general,  the  purpose  of  program  evaluation  is  to  assess  the  effectiveness  or  value  of  a  given 
program  or  experimental  project,  or  to  compare  alternative  programs.  In  addition  to  such 
summative  evaluation,  formative  program  evaluation  can  be  used  to  help  develop  new  programs 
(Grant,  1978).  Evaluation  efforts  may  reflect  a  broad  range  of  experimental  rigor  from  the  one- 
shot,  post  hoc  case  study  approach  in  which  a  single  program  is  examined  after  a  program  or 
treatment  has  been  given  to  fully  planned  approaches  based  on  the  principles  of  research  design 
(Campbell  &  Stanley,  1963).  Further,  evaluation  studies  may  also  include  an  assessment  of  the  cost 
effectiveness  or  cost  benefit  of  the  program.  However,  social  programs,  including  quality  of  life 
programs,  are  difficult  to  evaluate  in  this  manner  due  to  the  many  intangible  costs  and  benefits 
often  associated  with  them  (Lorenson,  1977,  p.  8). 

To  provide  an  understanding  of  program  evaluation,  a  discussion  of  the  role  of  measurement 
in  evaluation  and  an  overview  of  the  evaluation  process  will  be  presented. 

Measurement 

The  key  to  the  evaluation  process  is  measurement.  Anecdotal  information,  which  may  provide 
richness  to  any  evaluation,  is  not  in  and  of  itself  sufficient  for  adequate  evaluation.  While  program 
managers  or  workers  may  believe  their  program  is  having  a  positive  effect  and  may  cite  individual 
clients  who  benefited  from  the  program,  proper  evaluation  requires  measurable  evidence  of 
program  outcomes.  A  wide  variety  of  measures  can  be  used  in  program  evaluation.  These  measures 
include  performance  (i.e.,  program  indicators),  effectiveness,  and  efficiency. 

Measures  of  Performance.  Measures  of  performance  reflect  the  processes  underlying  the 
program  without  attaching  any  evaluation  to  the  measures.  They  measure  program  activity  rather 
than  accomplishments  or  progress  toward  goals.  For  example,  measures  of  performance  for  an 
alcohol  rehabilitation  program  might  include  number  of  inpatients  treated;  service  members 
requesting  inpatient  service;  outpatients  seen;  visits  per  outpatient,  size  of  the  staff,  and  operating 
hours.  These  measures  reflect  the  work  being  done  by  the  rehabilitation  program,  but  do  not 
indicate  how  well  the  program  is  working. 
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Measures  of  Effectiveness.  Measures  of  effectiveness,  on  the  other  hand,  assess  the  extent  to 
which  a  program  meets  its  goals.  Measures  of  effectiveness  reflect  the  impact  of  the  program  on 
its  users  and  the  degree  to  which  it  meets  identified  needs.  Because  programs  typically  have  many 
goals  (e.g.,  curing  identified  alcoholics,  preventing  relapses,  limiting  the  number  of  new 
alcoholics,  reducing  DWI  citations,  reducing  alcohol-related  accidents),  multiple  measures  of 
effectiveness  are  often  used.  Many  measures  of  effectiveness  are  expressed  as  ratios.  For  the 
alcohol  rehabilitation  program,  such  measures  might  include  percent  of  clients  successfully 
returned  to  the  fleet  or  the  recidivism  rate  among  treated  individuals.  In  addition,  client  satisfaction 
with  services  provided  should  also  be  considered  a  measure  of  effectiveness.  Further,  if  a  goal  for 
the  indicator  exists  or  can  be  developed,  program  indicators,  such  as  those  listed  in  the  previous 
paragraph,  can  be  converted  to  measures  of  effectiveness.  For  example,  if  a  stated  goal  of  the 
alcohol  rehabilitation  program  is  to  provide  inpatient  services  to  all  who  request  it,  then  an 
appropriate  measure  of  effectiveness  would  be  the  percent  of  service  members  requesting  service 
who  are  actually  provided  inpatient  service. 

In  addition  to  meeting  the  many  goals  related  to  providing  appropriate  service  to  clients,  social 
programs  must  also  meet  requirements  placed  on  the  program  by  program  managers,  parent 
organizations,  or  congress;  follow  appropriate  legislation  or  other  applicable  regulations;  remain 
competitive  with  other  similar  programs;  maintain  a  qualified  and  satisfied  staff;  and  so  on.  These 
additional  goals  can  also  be  included  in  program  evaluation.  Compliance  evaluations,  for  example, 
address  whether  a  program  and  its  expenditures  are  in  compliance  with  applicable  law,  regulations, 
and  procedures  (General  Accounting  Office,  1978). 

Measures  of  Efficiency.  Finally,  measures  of  efficiency  are  input-to-output  ratios  that  track 
inputs  or  resources  consumed  by  the  program  (e.g.,  the  costs  related  to  labor  hours,  facilities  and 
materials,  energy,  etc.)  in  relation  to  the  outputs  produced  (e.g.,  problem-drinkers  treated  or 
“cured”  alcoholics).  These  measures  determine  whether  the  program’s  resources  are  used 
efficiently  and  economically.  Although  program  sponsors  may  prefer  to  emphasize  the  “positive” 
side  of  evaluation  (program  effectiveness),  they  must  not  ignore  the  “negative”  side  (program 
costs).  Responsible  evaluations  will  always  include  an  assessment  of  costs  in  the  evaluation 
(Windle  &  Sharfstein,  1978).  A  measure  of  efficiency  for  the  alcohol  rehabilitation  program  might 
be  the  cost  per  client  treated. 

Cost-benefit  and  cost-effectiveness  assessments  are  specific  types  of  measures  of  efficiency: 
Both  techniques  are  comparative  in  nature  and  are  designed  for  use  in  evaluating  alternatives 
(Levin,  1987).  Both  cost-benefit  and  cost-effectiveness  techniques  measure  costs  in  the  same 
way-by  specifying  all  of  the  resources  required  by  the  program  and  assessing  their  market  value. 
However,  cost-benefit  and  cost-effectiveness  techniques  differ  in  how  they  measure  outcomes. 
Cost-benefit  analysis  is  used  when  the  benefits  can  be  assessed  in  monetary  terms  as  the  costs  are. 
Educational  and  training  programs  can  be  evaluated  with  this  method  because  they  are  intended  to 
improve  employment  or  earnings  of  participants.  The  measurable  benefits  are  the  increased 
earnings  due  to  program  participation.  Of  several  programs  under  consideration,  the  program  that 
produces  the  greatest  increase  in  earning  potential  with  the  smallest  investment  of  resources  would 
have  the  best  cost-benefit  ratio.  With  some  effort,  a  variety  of  outcomes  of  social  programs  can  be 
translated  to  monetary  values.  The  value  of  a  “cured”  alcoholic  can  be  estimated  by  the  dollar 
savings  accrued  to  the  organization  by  returning  a  trained  sailor  to  the  fleet  rather  than  recruiting 
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and  training  a  new  sailor.  Cost-benefit  analysis  is  very  useful  in  assessing  multiple  outcomes  of  a 
program  when  they  all  can  be  converted  into  dollar  values. 

However,  when  the  outcomes  cannot  be  easily  converted  into  market  outcomes,  cost- 
effectiveness  analysis  is  appropriate.  Here,  alternative  interventions  or  programs  are  evaluated 
against  a  common  set  of  goals.  For  example,  training  programs  may  be  evaluated  by  testing 
participants  on  their  knowledge  and  understanding  of  course  content.  The  option  that  requires  the 
fewest  resources  to  meet  a  particular  objective  (e.g.,  competency  score)  would  have  the  best  cost- 
effectiveness  ratio.  In  comparing  alternative  alcohol  education  courses,  the  one  that  resulted  in  the 
best  end-of-course  grades  on  an  alcohol-knowledge  test  with  the  fewest  costs  invested  would  be 
the  most  cost-effective.  Likewise,  in  comparing  different  alcohol  rehabilitation  programs,  the  one 
that  had  the  best  success  rate  (i.e.,  percent  of  clients  successfully  treated)  with  the  smallest  costs 
would  be  most  cost-effective  for  that  goal.  A  current  challenge  for  the  field  of  evaluation  is 
developing  means  to  combine  multiple  outcomes  when  they  cannot  be  converted  into  dollar  values. 
Utility  analysis  is  one  approach  that  has  been  used.  In  this  technique,  the  independent  outcomes  are 
converted  to  ratings  on  utility  scales  that  share  common  measurement  properties.  The  separate 
scales  can  then  be  aggregated  into  a  total  utility  index  (Keeny  &  Raiffa,  1976;  Levin,  1987). 

Program  evaluation  may  include  any  or  all  of  these  types  of  measures.  In  fact,  many 
evaluations  rely  on  multiple  measures  to  assess  social  programs.  Few  programs  can  be  effectively 
evaluated  with  a  single  indicator  or  effectiveness  measure.  Likewise,  cost-benefit  and  cost- 
effectiveness  evaluations  should  consider  all  costs  and  all  outcomes.  If  an  alcohol  treatment 
program  had  an  impressive  initial  success  rate,  but  also  had  the  highest  recidivism  rate,  it  could  not 
be  considered  most  cost-effective  overall. 

An  additional  consideration  in  conducting  any  evaluation,  particularly  a  cost  effectiveness 
study,  is  the  cost  of  the  evaluation  itself.  While  a  “quick-and-dirty”  look  at  the  operation  of  a 
program  may  incur  few  costs,  a  well-planned  systematic  evaluation  will  involve  a  substantial 
investment.  For  example,  in  fiscal  year  1977,  the  Department  of  Housing  and  Urban  Development 
devoted  18  staff-years  and  $2.3  million  to  evaluating  its  programs  (General  Accounting  Office, 
1978).  Agencies  considering  evaluation  efforts  must  balance  the  goals  for  the  evaluation  and  the 
scope  of  the  evaluation  with  the  investment  they  can  make  in  the  effort. 

There  are  several  important  characteristics  of  a  good  measurement  package  (Mohr,  Shumate, 
&  Magnusson,  1983).  First,  measures  should  focus  on  the  key  objectives  of  the  program.  Thus, 
prior  to  program  evaluation,  clear  statements  of  program  goals  must  be  developed.  Goals  may  be 
derived  from  legislation,  compliance  manuals,  written  departmental  or  program  instructions, 
national  norms,  or  program  managers’  objectives  for  the  program.  Second,  measures  should  be 
objective  and  definable.  This  is  particularly  important  when  independent  sites  are  to  be  evaluated 
or  when  multiple  evaluators  contribute  information  about  the  program.  If  measures  are  calculated 
even  slightly  differently  in  each  site,  there  will  be  no  common  basis  for  comparison  or  aggregation. 
In  certain  circumstances,  subjective  measures  may  supplement  objective  measures  (e.g.,  to  judge 
client  satisfaction  with  services  provided).  Third,  measures  should  be  complete  and 
comprehensive.  Because  all  important  goals  of  the  program  must  be  assessed,  multiple  measures 
are  often  required.  If  all  goals  are  not  included  in  the  measurement  package,  the  program  may 
appear  to  be  successful  on  measured  goals  at  the  expense  of  unmeasured  goals.  Fourth,  measures 
should  be  accessible.  They  should  not  be  so  difficult  to  obtain  that  the  potential  benefits  of  the 
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information  are  negated  by  the  costs  required  to  obtain  the  information.  Existing  data  collection 
procedures  (e.g.,  management  information  systems  and  other  automated  record  keeping 
mechanisms)  should  be  utilized  whenever  possible. 

An  Overview  of  Evaluation 

Figure  1  presents  an  overview  of  the  evaluation  process.  At  the  left  are  shown  the  factors 
driving  quality  of  life  programs:  need  or  demand,  goals,  and  resources.  To  conduct  a  proper 
evaluation  of  a  quality  of  life  program,  these  must  be  identified  and  fully  understood.  The  need  to 
explicitly  define  all  program  goals  cannot  be  unde  estimated.  If  a  goal  is  not  specified,  it  cannot  be 
measured  and  may  be  overlooked  in  the  evaluation  analysis. 

First-stage  evaluation  assesses  the  outcomes  of  the  quality  of  life  program  in  three  areas:  goal 
attainment,  attitudinai  enhancement,  and  tangible  improvements.  Evaluation  of  goal  attainment 
relies  on  measures  of  effectiveness  defined  above.  It  may  also  include  assessment  of  the  extent  to 
which  the  program  reaches  its  intended  client  base.  Measures  of  performance  and  anecdotal 
evidence  of  program  outcomes  can  enrich  this  area  of  evaluation.  Evaluation  of  attitudinai 
enhancement  can  supplement  the  goal  ttainment  evaluation.  If  attitudinai  improvement  such  as 
increased  job  satisfaction  or  greater  spousal  support  for  the  service  member’s  career  are  part  of  the 
program’s  goals,  these  can  be  included  in  goal  attainment  evaluation,  as  well.  Finally,  evaluation 
of  tangible  improvements  provides  evidence  of  the  program’s  impact  on  “bottom-line”  factors  such 
as  retention,  job  performance,  discipline  rates,  absenteeism,  and  readiness. 

Second-stage  evaluation  attempts  to  combine  the  many  results  found  in  first-stage  evaluation 
either  through  cost-benefit  analysis  (if  all  outcomes  can  be  translated  to  dollar  values)  or  through 
cost-effectiveness  analysis  (if  they  cannot).  While  there  is  substantial  evidence  that  job 
satisfaction,  spousal  support  for  one’s  career,  and  other  attitudinai  factors  can  have  a  positive  effect 
on  outcomes  such  as  retention  (through  career  intent),  absenteeism,  and  job  performance  (e.g., 
Bruce  &  Burch,  1989;  Mohr,  Holzbach,  &  Morrison,  1981;  Porter  &  Steers,  1973;  Sterling  & 
Allen,  1983),  it  is  difficult  to  include  this  link  in  cost-effectiveness  analysis.  Mirvis  and  Lawler 
(1977)  presented  an  exploratory  approach  that  can  be  used  to  attach  behavioral  costs  to  attitudinai 
factors.  Their  approach  treats  attitudinai  measures  as  indicators  of  subsequent  behavior.  This 
methodology  suggests  that  including  attitudinai  factors  in  cost-benefit  analysis  may  eventually 
feasible. 

Before  beginning  an  evaluation  effort,  many  factors  must  be  specified.  These  include  the  time 
frame  for  evaluation,  level  of  analysis,  and  purpose  of  the  evaluation. 

Time  Frame  for  Evaluation.  The  time  frame  encompassed  by  an  evaluation  can  vary  widely. 
Program  evaluation  may  occur  as  a  one-shot  study  at  the  conclusion  of  a  temporary  program  (e.g., 
a  program  to  provide  emergency  aid  after  a  natural  disaster).  More  likely,  it  will  be  ongoing, 
providing  continual  information  about  the  effectiveness  of  a  program  or  set  of  programs.  Ongoing 
evaluations  might  feasibly  provide  monthly  updates  about  program  effectiveness.  More  likely, 
year-end  evaluations  occur. 
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Level  of  Analysis.  In  planning  an  evaluation,  the  appropriate  level  of  analysis  for  performance 
measures  must  also  be  determined.  The  scope  of  the  analysis  as  a  whole  might  be  at  the  program 
site  level,  program  level,  or  program  area.  Program  site  evaluation  would  provide  information 
about  the  effectiveness  of  individual  service-providing  sites  (e.g.,  comparing  effectiveness  of 
individual  Family  Service  Centers).  Thus,  the  performance  of  sites  could  be  compared  to  identify 
more  effective  programs.  Program  level  analysis  would  evaluate  the  effectiveness  of  all  service¬ 
providing  sites  as  a  whole  (e.g.,  all  Family  Service  Centers).  This  type  of  evaluation  would  reflect 
the  effectiveness  of  the  Family  Service  system  Navy- wide.  Finally,  program  area  evaluation  would 
evaluate  the  broad  functional  areas  comprising  the  quality  of  life  arena  (e.g.,  housing,  personal 
support,  etc.).  Such  a  global  analysis  is  much  more  difficult  because  of  the  many  programs  with 
unique  goals  and  outcomes  that  are  found  within  each  functional  area. 

Purpose  of  the  Evaluation.  Finally,  the  purpose  of  the  evaluation  must  be  specified  clearly.  It 
will  help  drive  the  scope  and  focus  of  the  evaluation  effort.  Evaluations  can  be  done  to:  compare 
program  effectiveness  from  year-to-year,  from  site-to-site,  or  across  services;  to  evaluate  different 
treatment  methods,  program  structures,  or  benefit  packages;  determine  customer/client  satisfaction 
with  services  provided;  or  determine  that  tax  dollars  are  being  used  effectively.  Each  of  these  will 
require  different  methods  and  different  measures. 

LITERATURE  REVIEW 

The  results  of  a  literature  review  conducted  to  identify  previous  efforts  to  evaluate  quality  of 
life  programs  will  be  summarized  by  program  area.  Within  each  area,  the  studies  found  will  be 
cited  along  with  a  brief  discussion  of  the  strengths  and/or  weaknesses  of  the  studies,  their  results, 
and  their  applicability  to  Navy  evaluation  efforts.  Given  the  time  available  to  conduct  this  review, 
it  cannot  be  considered  exhaustive. 

In  reviewing  the  literature  available  in  this  area,  it  should  be  kept  in  mind  that  negative  or  no¬ 
effect  results  are  not  often  published  in  journals,  books,  or  technical  reports.  Thus,  most  available 
literature  indicates  that  whatever  program  was  implemented  was  effective.  Furthermore,  as  Bowen 
and  Scheirer  (1986)  note,  “rigorous  evaluation  of  programs  and  services  is  comparatively  rare 
within  the  military”  (p.  196).  This  review  includes  military  program  evaluations  found,  as  well  as 
applicable  evaluations  from  civilian  programs.  Within  the  quality  of  life  arena,  anecdotal  reports 
and  “how-to”  guidelines  abound.  Sound  evaluation  is  rare.  Finally,  the  purpose,  scope,  and  design 
of  specific  quality  of  life  programs  vary  over  time,  as  does  the  surrounding  economic  environment. 
Assuming  that  results  of  previous  evaluation  efforts  automatically  apply  to  today’s  programs  is 
inappropriate. 

Compensation 

Within  the  compensation  functional  area,  few  evaluation  studies  were  identified.  Those  cited 
below  address  compensation  and  other  pay,  retirement  benefits,  bonuses,  and  relocation 
reimbursement.  In  examining  studies  presented  below,  care  should  be  taken  to  interpret  all  findings 
in  light  of  the  time  period  in  which  they  were  conducted.  Changing  unemployment  rates  and  other 
economic  indicators  may  invalidate  the  findings  of  past  studies. 
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Regular  Military  Compensation 

The  General  Accounting  Office  (GAO)  (1986a)  analyzed  the  comparability  of  military  and 
civilian  compensation.  They  compared  the  compensation  of  all  military  and  all  civilian  workers 
who  were  employed  full  time  in  calendar  year  1984  and  who  met  certain  other  matching  criteria 
(e.g.,  age,  educational  attainment,  etc.).  Military  compensation  was  defined  as  Regular  Military 
Compensation  (RMC)  plus  special  and  incentive  pays.  RMC  is  comprised  of  basic  pay,  allowances 
for  quarters  and  subsistence,  and  a  DoD-calculated  imputed  tax  advantage  for  service  members 
receiving  cash  allowances.  Enlistment  and  retention  bonuses  were  not  included.  Civilian 
compensation  was  defined  as  wages  and  salaries  reported  in  the  March  1985  Current  Population 
Survey  conducted  by  the  Bureau  of  the  Census.  Benefits  were  valued  as  the  cost  to  the  employer 
of  providing  the  benefit.  Benefits  included  health  and  life  insurance,  survivors’  benefits,  and 
retirement.  Results  of  the  comparison  showed  that  military  compensation  was  higher  than  that  of 
comparable  civilians  in  all  cases  except  that  of  male  high  school  graduates  of  certain  ages. 

In  separate  analyses,  GAO  also  attempted  to  determine  the  relationship  between  military- 
civilian  compensation  comparability  and  recruiting  and  retention.  GAO  concluded  that  many 
factors  in  addition  to  compensation  contribute  to  the  military’s  ability  to  recruit  adequate 
personnel.  These  factors  include:  improved  post-service  educational  benefits,  more  recruiting 
resources,  and  lower  unemployment  rates.  They  also  concluded  that  no  clear  relationship  could  be 
found  between  differences  in  military  and  civilian  compensation  and  military  retention. 

Another  GAO  report  (1986b)  also  examined  the  compensation  comparability  issue.  In  contrast 
to  the  above  report,  which  compared  earnings  on  an  “age-earnings”  basis,  the  present  study 
compared  earnings  for  similar  occupations,  skill  levels,  experience,  and  responsibilities.  GAO 
points  out  that  this  type  of  comparison  is  limited  because  many  military  occupations  have  no 
civilian  counterpart,  military  service  is  unique,  and  it  provides  fringe  benefits  not  often  found  in 
civilian  employment  situations.  They  were  able  to  compare  compensation  for  a  sample  of  52 
occupations  covering  about  4  percent  of  the  enlisted  force.  Results  of  the  comparison  showed  that 
military  compensation  for  these  occupations  was  generally  lower  than  civilian  compensation. 
However,  these  occupations  were  primarily  technical  in  nature,  involving  highly-skilled  work 
(e.g.,  computer  operators  and  technicians  of  various  sorts)  and  tend  to  be  very  well  paid  in  the 
private  sector. 

Haber  and  Stewart  (1975)  examined  the  Navy  retention  effects  of  the  1971  military  pay 
increase.  They  used  a  measure  of  pay  elasticity  for  the  analyses.  Pay  elasticity  is  an  estimate  of  the 
sensitivity  of  the  reenlistment  rate  to  changes  in  military  pay  and,  hence,  provides  a  cost-benefit 
measure  of  the  effectiveness  of  pay  increases  in  improving  reenlistment  rates.  Looking  at  first-term 
reenlistment,  they  found  that  pay  elasticity  was  high.  In  almost  one-half  of  the  cases,  a  1  percent 
increase  in  pay  resulted  in  a  3  percent  increase  in  reenlistment.  No  differences  were  found  by 
occupational  area. 

Institute  for  Research  Studies,  Inc.  (1979)  developed  a  model  to  determine  the  extent  to  which 
compensation  influences  retention  of  personnel  in  various  Navy  ratings.  Results  of  the  analysis  led 
them  to  conclude  that  “even  a  partial  shift  in  the  amount  of  compensation  from  Regular  Military 
Compensation  (RMC)  to  Basic  Pay  would  result  in  greater  retention  of  enlisted  personnel”  (p.  3). 
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Kostiuk  (1985)  studied  the  effects  of  pay  on  the  retention  of  Marine  Corps  aviators.  He  used  a 
simplified  version  of  the  annualized  cost  of  leaving  (ACOL)  model  to  estimate  the  impact  of  pay 
changes  on  the  voluntary  turnover  of  Marine  aviators.  By  comparing  the  value  of  continuing  a 
military  career  to  the  value  of  a  civilian  career  (as  a  civilian  airline  pilot),  he  concludes  that  pay  has 
a  significant  effect  on  aviator  retention. 

Retirement  Benefits 

Retirement  pay  is  a  key  element  within  the  compensation  arena.  Because  20  years  of  service  is 
required  to  receive  retirement  pay,  it  is  believed  to  be  a  strong  retention  factor  for  personnel 
between  10  and  20  years  of  service.  Retention  rates  drop  sharply  after  10  years  (Warner,  1979). 
Because  of  the  cost  associated  with  this  system,  alternative  systems  are  periodically  proposed. 
Studies  found  within  this  area  do  not  evaluate  current  retirement  policies.  Rather,  they  describe 
models  developed  to  assess  the  effects  of  alternative  retirement  systems  of  force  retention.  Instead 
of  presenting  results  of  specific  alternatives  evaluated  in  these  reports,  the  models  will  be 
identified.  Users  wishing  to  examine  any  particular  retirement  system  can  do  so  using  these 
models. 

To  evaluate  four  specific  proposed  alternative  retirement  systems,  the  Center  for  Naval 
Analyses  developed  an  economic  model  to  evaluate  quantitatively  the  impact  that  alternatives 
would  have  on  retention  and  the  average  experience  levels  of  the  armed  forces  (Warner,  1979). 
This  model  is  a  variant  of  the  cost  of  leaving  model  proposed  by  Gotz  and  McCall  (1977).  The 
report  describes  the  application  of  the  model  to  the  four  alternative  retirement  systems. 

Likewise,  Chipman  and  Silverman  (1978)  and  Chipman  and  Mumm  (1978)  developed  a  model 
for  forecasting  Navy  enlisted  retention  rates  and  service  continuation  rates  under  alternative 
retirement  systems.  Their  technique  was  based  on  the  work  of  Gotz  and  McCall  and  can  also  be 
used  to  examine  sub-groups  within  the  Navy  that  have  homogeneous  occupations  and  retention 
behavior. 

Special  Pay  and  Bonuses 

Reenlistment  bonuses  and  other  special  pay  categories  also  have  an  important  role  in  the 
overall  military  compensation  package.  They  help  to  prevent  shortages  in  critical  occupations  and 
can  be  targeted  where  they  are  needed  most.  Hosek  and  Peterson  (1985)  examined  the 
effectiveness  of  bonuses  when  the  method  of  bonus  payment  was  changed  from  installment  to 
lump  sum  in  April  1979.  Inputs  to  their  analysis  include  reenlistment,  extension,  and  retention  data 
for  each  occupation;  bonus  coverage  and  amount;  military/civilian  wage  index  variable; 
unemployment  rate;  educational  background  of  enlisted  personnel;  and  the  percent  black.  They 
found  that  both  types  of  reenlistment  bonuses  increase  reenlistment  and  retention  rates  and 
decrease  extension  rates.  Furthermore,  they  found  that  lump  sum  bonuses  were  more  cost-effective 
than  installment  bonuses. 

The  forecasting  model  developed  by  Chipman  and  Mumm  (1978)  can  also  be  used  to  study  the 
effects  of  alternative  economic  incentives  at  various  points  of  time  in  a  sailor’s  career  on 
continuance  rates.  However,  they  do  not  present  any  specific  analyses  along  this  line. 
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Weber  (1987)  reviewed  the  effects  of  the  Navy’s  aviation  continuation  bonus  (Aviation  Officer 
Continuation  Pay  or  AOCP)  implemented  in  July  1 98 1  to  address  severe  retention  problems  among 
naval  aviators.  AOCP  is  paid  to  a  naval  aviator  to  extend  his  period  of  obligated  service  for  a 
certain  length  of  time.  In  the  first  2  months  after  implementation,  74.3  percent  of  the  aviators 
eligible  for  the  bonus  had  applied  for  it.  However,  Weber’s  analysis  revealed  that  49  percent  of 
those  applying  for  the  bonus  were  in  their  1 1th  to  15th  year  of  service  and  reasonably  could  have 
been  expected  to  continue  their  careers  even  without  a  bonus.  Other  problems  were  identified  and 
subsequent  changes  to  the  AOCP  program  were  summarized.  Weber  concludes  that  while  bonuses 
can  have  a  significant  effect  on  retention  (and  the  current  version  of  AOCP  does),  the  bonus  was 
originally  targeted  at  the  wrong  group  at  the  wrong  time. 

Kostiuk  (1985)  evaluated  the  effects  of  three  proposed  changes  to  the  Marine  Corps  Aviation 
Career  Incentive  Pay  (ACIP).  He  calculated  the  costs  and  benefits  of  each  proposal  using  the 
ACOL  model.  Benefits  are  the  reduced  ACIP  payments  and  costs  are  the  expense  of  training  new 
pilots  to  replace  those  who  leave  due  to  the  pay  change.  All  three  proposed  changes  provide  a  net 
increase  in  total  expenditures  over  the  existing  system. 

Finally,  the  Naval  Aviator  Active  Duty  Service  Obligation  (ADSO)  Study  Group  (1988) 
analyzed  ACEP  (flight  pay)  and  AOCP  combinations  in  an  attempt  to  determine  a  cost-effective 
compensation  package  for  pilots.  The  group  provides  a  historical  summary  of  both  programs.  The 
ACOL  model  was  used  to  evaluate  alternatives  and  make  recommendations  for  changing  the 
current  bonus  systems. 

Travel  and  Transportation  Reimbursements 

The  military  is  probably  the  largest  employee-mover  in  the  country  and,  as  such,  provides 
many  services  in  the  areas  of  reimbursement  and  relocation  assistance.  While  relocation  assistance 
takes  many  forms  in  military  and  civilian  organizations,  no  quantitative  studies  of  the  effectiveness 
of  these  programs  were  found.  At  most,  personnel  journals  suggest  that  organizations  must  have 
adequate  relocation  assistance  programs  to  limit  the  extent  to  which  employees  refuse  moves  (by 
either  staying  in  their  current  location  or  leaving  the  company).  Organizations  must  also  limit  the 
stress  of  relocation  on  employees  and  their  families.  No  attempts  to  assess  refusal  rates  under 
different  relocation  programs  were  found. 

Relocation  costs  have  risen  substantially  in  the  recent  past.  Milbrandt  (1983)  found  that  the 
average  cost  of  relocation  rose  from  $7,500  in  1968  to  $37,500  in  1983.  This  increase  is  due  to 
increased  costs  and  increased  services.  Civilian  organizations  now  offer  a  wide  range  of  services 
in  their  relocation  package  (Debats,  1982;  Milbrandt,  1983).  These  may  include  shipment  of 
household  goods;  home  selling,  finding,  or  servicing;  house  hunting  trips;  temporary  living 
expenses;  living  cost  differentials;  mortgage  rate  buy  downs;  and  spouse  employment  assistance. 

Milbrandt  (1983)  discusses  ways  in  which  relocation  costs  can  be  managed  or  constrained. 
However,  these  have  very  limited  applicability  to  military  relocations.  He  suggests  reducing  the 
number  of  relocations,  avoiding  high-cost  areas,  considering  short  distance  relocations  before  long 
distance  moves,  and  limiting  the  transfers  of  homeowners. 
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The  Air  Command  and  Staff  College  (1986)  conducted  a  study  to  compare  reimbursement  for 
permanent  change  of  station  (PCS)  moves  provided  to  uiilitary  members  and  federal  civilian 
employees.  Their  analysis  was  based  on  a  1984  survey.  Results  showed  that,  excluding  home 
ownership  costs,  military  members  must  absorb  75  percent  of  out-of-pocket  expenses  related  to  the 
move.  On  the  other  hand,  many  of  these  out-of-pocket  expenses  are  reimbursed  for  federal  civilian 
employees.  The  authors  conclude  that  military  members  are  not  receiving  comparable 
reimbursement  for  their  PCS  moves. 

Hansen  and  Handforth  (1980)  also  compared  compensation  for  PCS  moves  among  the  military, 
civil  service,  and  private  sector.  Using  a  set  of  standard  relocation  scenarios,  they  estimated  actual 
costs  for  each  move  as  well  as  the  standard  reimbursements  provided  by  military,  civil  service,  and 
private  sector  employers.  They  found  significant  differences  in  reimbursements.  (Note  that  all 
comparisons  are  based  on  1979  costs  and  reimbursement  coverages.)  Over  the  six  standard 
relocation  scenarios,  they  found  that  military  members  pay  an  average  of  $9,368  in  out-of-pocket 
expenses;  corporate  employees  pay  an  average  of  $1,254  in  out-of-pocket  expenses;  and  civil 
service  employees  are  reimbursed  an  average  of  $1,058  in  excess  of  their  actual  expenses.  Major 
discrepancies  resulting  in  the  large  military  out-of-pocket  expenses  are:  military  members  are  not 
reimbursed  for  house  hunting  trips  or  temporary  lodging  either  at  the  old  or  new  duty  station,  nor 
do  they  receive  any  assistance  in  purchasing  a  home  if  government  quarters  are  unavailable. 
Reimbursement  rates  for  miscellaneous  expenses  and  travel  costs  are  also  significantly  different. 
Military  service  members  do  have  better  coverage  in  two  areas:  a  greater  weight  limit  on  household 
goods  and  temporary  storage  of  goods  for  up  to  180  days.  Finally,  the  authors  point  out  that  these 
comparisons  are  valid  for  military  E-5s  and  up.  E-4s  and  below  are  not  eligible  for  household 
goods  shipment  over  225  pounds,  mileage  allowance  for  dependents,  or  dislocation  allowance  (for 
incidental  expenses).  These  out-of-pocket  expenses  represent  a  significant  reduction  in  the  real  pay 
service  members  receive  and  may  have  implications  for  turnover  and  retention. 

Health  Care  and  Promotion 

Within  the  scope  of  the  medical  and  mental  health  care  and  prevention  functional  area  fall 
many  inter-related  quality  of  life  programs.  These  will  be  addressed  separately  in  this  section. 
These  programs  include  health  care  systems,  health  and  physical  readiness  programs,  and  alcohol 
and  drug  abuse  control  programs. 

Health  Care  Systems 

Adequate  evaluations  have  not  been  conducted  due  to  the  many  recent  changes  in  the  military 
health  care  system  (e.g.,  CHAMPUS  and  CHAMPUS-Prime).  While  the  popular  press  abounds 
with  horror  stories  about  the  state  of  military  medical  care,  these  do  not  represent  adequate 
evaluation  efforts  and  will  not  be  included  here.  Few  evaluations  were  found  in  the  area  of  medical 
care  itself.  Kozlowski  and  Oleksy  (1987)  discuss  ways  to  assess  the  cost-effectiveness  of  an 
organization’s  benefit  plan  (including  medical  care)  and  provide  suggestions  and  alternatives  to 
lower  such  costs.  Several  evaluation  efforts  were  found  in  the  mental  health  care  arena. 

Kaplan  and  Bush  (1982)  proposed  a  measure  of  health  status  that  can  be  used  in  evaluation 
efforts.  The  “Well-Year”  expresses  the  output  of  health  programs  in  terms  of  the  number  of  added 
years  of  life  and  the  health-related  “quality  of  life”  of  those  years  produced  by  a  treatment. 


10 


program,  or  health  care  system.  Once  calculated,  the  Well- Years  can  be  divided  into  the  cost  of  the 
program  to  determine  the  cost-utility  of  the  program.  On  this  basis,  alternative  programs  or 
treatments  can  be  compared.  Kaplan  and  Bush  provide  a  sample  calculation  of  Well-Years  for 
illustrative  purposes. 

Hall  and  Doman  (1988)  reviewed  available  studies  on  consumer  satisfaction  with  medical  care. 
They  used  meta-analytic  techniques  to  derive  relative  levels  of  satisfaction  with  1 1  aspects  of 
medical  care  over  all  studies.  No  information  was  given  describing  the  types  of  medical  services 
evaluated.  In  the  studies  reviewed,  consumers  were  most  satisfied  with  humaneness  and 
competence  and  least  satisfied  with  cost,  access,  bureaucracy,  and  attention  to  psychosocial 
problems.  It  appears  that  patients  are  relatively  satisfied  with  the  actual  care  they  are  receiving,  but 
are  less  satisfied  with  the  process  or  system  that  provides  the  services.  Hall  and  Doman  provide 
information  about  these  ratings  that  would  be  useful  for  tapping  consumer  satisfaction  with 
medical  services. 

Moore  (1989)  reviewed  health  care  cost  containment  efforts  in  the  public  sector.  In  1984, 
employers’  costs  for  medical  benefits  rose  to  $1,285  per  employee,  an  increase  of  1 15  percent  in  8 
years.  Private  sector  organizations  have  been  aggressively  pursuing  cost-containment  methods. 
Moore  sought' to  determine  the  extent  of  similar  efforts  in  the  public  sector.  Results  of  a  survey 
completed  by  the  benefit  directors  of  75  large  cities  across  the  United  States  reveal  that  public 
agencies  are  beginning  to  pursue  cost-containment  methods,  but  lag  behind  private  sector 
organizations  in  their  progress.  Tangible  benefits  are  limited  though,  largely  because  public 
agencies  lack  the  comprehensive  medical  costs  data  tracked  by  private  organizations. 

Moore  presents  six  major  strategies  used  to  contain  medical  costs:  (1)  self-insurance  or  self¬ 
funding  of  insurance,  (2)  cost-sharing  actions  such  as  increased  deductibles  and  co-insurance,  (3) 
claims  management  and  utilization  review,  (4)  financial  incentives  to  employees  to  lower  costs,  (5) 
alternative  delivery  systems  (e.g.,  health  maintenance  organizations  (HMOs)  and  preferred 
provider  organizations  (PPOs)),  and  (6)  health  promotion  and  wellness  programs.  Moore  also 
reviews  studies  citing  cost  savings  attributable  to  any  of  these  methods.  He  indicates  that  in  private 
sector  organizations,  estimated  cost-savings  from  self-funding  range  from  2  to  20  percent.  Cities 
responding  to  the  survey  who  could  evaluate  their  costs  savings  in  this  area  reported  10  to  20 
percent  savings.  A  Health  Research  Institute  study  (1983)  was  cited  that  reported  savings  of  6 
percent  of  total  claims  attributable  to  increased  use  of  deductibles  for  medical  care.  A  study  by 
Lillard,  Manning,  Peterson,  Lurie,  and  Goldberg  (1986)  found  that  people  who  had  to  pay  part  of 
their  medical  bills  themselves  had  one-third  fewer  visits  to  doctors  for  minor  problems  than  those 
whose  medical  care  was  fully  funded.  HMOs  and  PPOs  are  believed  to  be  effective  in  reducing 
medical  costs,  but  no  data  were  presented  to  support  this  position.  Finally,  Moore  reports 
substantial  savings  can  be  realized  through  health  promotion  programs.  (These  programs  will  be 
reviewed  below.) 

Prigmore  and  Crank  (1977)  propose  a  performance  index  that  can  be  used  within  the  military 
health  service  system.  It  provides  a  measure  of  the  relative  productivity  of  medical  care  facilities. 
The  authors  review  previous  research  in  the  area  of  health  care  performance  measurement  to 
identify  both  problems  with  previous  approaches  and  successful  measurement  methodology.  They 
present  an  extensive  review  of  Kaiser-Permanente’s  new  Cost  Center  Accountability  System 
(CCAS)  on  which  Prigmore  and  Crank’s  measurement  model  is  based.  CCAS  provides 
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management  information  in  the  areas  of  utilization,  performance,  membership,  access/backlogs, 
and  cost.  The  proposed  military  health  service  model  may  be  helpful  in  developing  standard 
measures  in  this  area. 

Academy  of  Health  Sciences  (1980)  evaluated  an  innovative  medical  screening  program  used 
with  Army  service  members.  Instead  of  using  the  traditional  morning  sick  call  procedures  to 
provide  outpatient  medical  care  to  active  duty  personnel,  the  Naval  Regional  Medical  Center 
(Orlando,  Florida)  developed  an  in-barracks  medical  screening  program.  Evaluation  showed  that 
recruits  participating  in  the  new  system  returned  to  active  duty  in  one-sixth  the  time  when 
compared  to  the  traditional  system.  Furthermore,  pharmaceutical  costs  were  reduced  by  two-thirds, 
total  visits  to  sick  call  were  reduced  by  20  percent,  and  85  percent  of  recruits  returned  to  duty 
without  requiring  the  services  of  a  physician  or  physician’s  assistant.  Because  fewer  recruits  were 
seen  at  the  branch  clinic,  waiting  times  there  were  reduced  as  well. 

Smith  (1975)  reviewed  several  previous  evaluations  of  mental  health  programs  and  reported  on 
the  comparison  of  a  community-oriented  regional  mental  health  center  to  a  traditional  state  hospital 
in  treating  seriously  disturbed  mental  patients.  Patients  were  tracked  for  3  years  after  beginning 
treatment  at  either  facility.  Hospitalization  rates  and  a  clearly  stated  measure  of  successful  patients 
were  used  for  evaluation.  Furthermore,  the  costs  related  to  each  treatment  program  were  compared. 
Although  the  community-oriented  approach  provided  more  humanitarian  care,  no  objective 
evidence  was  found  that  it  was  more  effective  in  decreasing  mental  disorder  disabilities.  Based  on 
the  evaluation  results,  the  focus  of  the  regional  mental  health  center  was  changed  substantially  to 
stress  client  rehabilitation. 

Kiresuk  and  Lund  (1975)  developed  a  measurement  technique  to  subject  mental  health  therapy 
to  evaluation  methodology.  The  “goal  attainment  scaling”  procedure  uses  pre-specified  anchors 
against  which  to  judge  clinical  progress  at  the  client,  therapist,  or  program  level.  Goal  attainment 
scales  are  similar  to  behaviorally  anchored  rating  scales  in  that  points  on  the  scale  are  anchored  to 
observable  behaviors.  The  authors  also  present  a  methodology  to  quantitatively  weight  and 
aggregate  scales  for  independent  outcomes  or  activities. 

Weisbrod  and  Helming  (1980)  describe  the  use  of  economic  cost-benefit  analysis  in  evaluating 
an  experiment  in  treating  the  mentally  ill.  The  experiment,  Training  in  Community  Living  (TCL), 
de-emphasized  hospitalization  and  made  community-based  resources  available  to  patients. 
Patients  seeking  treatment  were  assigned  randomly  to  the  existing  treatment  program  or  to  the  TCL 
program.  Evaluation  was  conducted  on  65  experimental  and  65  control  subjects  for  12  months 
following  their  admission.  They  found  that  the  experimental  program  was  10  percent  more  costly 
during  the  first  year  than  the  traditional  approach  ($8,093  vs.  $7,296  per  patient).  However, 
looking  at  the  economic  benefits  of  the  program,  the  authors  found  that  TCL  patients  had  greater 
labor  market  productivity  ($1,200  more  in  wages  per  person  per  year)  than  control  patients.  Other 
non-economic  benefits  of  the  TCL  program  were  that  patients  reported  greater  life  satisfaction,  had 
opened  more  savings  accounts,  had  purchased  more  insurance,  and  had  worked  more  days  during 
the  year.  The  authors  provide  a  thorough  discussion  of  the  use  of  cost-benefit  analysis  with  social 
service  programs  and  stress  the  need  to  assess  intangible  outcomes  as  well. 

Finally,  Hagedom,  Beck,  Neubert,  and  Werlin  (1976)  present  a  handbook  for  use  in  developing 
program  evaluation  capabilities.  The  handbook  compiles  a  variety  of  techniques  and  approaches  to 
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program  evaluation  that  are  applicable  to  community  mental  health  centers  and  many  other  quality 
of  life  programs.  They  include  sections  on  outcome  studies,  cost-effectiveness  analysis,  evaluation 
of  consultation  and  education  efforts,  and  assessment  of  quality.  Furthermore,  the  authors  provide 
estimates  of  the  level  of  effort  required  to  utilize  each  methodology. 

Health  and  Physical  Readiness 

Health  promotion  programs  (HPPs)  and  Employee  Assistance  Programs  (EAPs)  are 
contemporary  innovations  in  the  U.S.  workplace.  Few  have  been  in  existence  more  than  15  years 
and  most  are  less  than  5  years  old  (Roman  &  Blum,  1988).  While  the  terms  are  somewhat  similar, 
there  is  a  basis  for  distinction.  HPPs  use  a  proactive,  “primary  prevention  strategy”  to  decrease  the 
risks  of  health  problems  that  are  costly  to  the  employee  and  the  employer.  EAPs  use  a  reactive, 
“secondary  prevention  strategy”  to  provide  early  treatment  for  symptoms  that  have  already 
emerged,  thereby  limiting  the  costs  to  employee  and  employer.  HPPs  typically  focus  on  weight 
reduction,  smoking  cessation,  stress  management,  physical  fitness,  and  dietary  changes.  EAPs 
focus  on  alcohol  and  drug  problems  and  emotional  or  family  problems.  Both  are  undertaken  in  an 
attempt  to  reduce  health  care  costs  and  improve  job  performance. 

The  military  quality  of  life  system  has  no  single  program  that  corresponds  directly  to  either  of 
these  private-sector  programs.  Many  of  the  factors  included  in  HPPs  and  EAPs  are  included  in 
Navy  quality  of  life  efforts,  but  they  are  not  grouped  under  similar  umbrella  programs.  While  most 
of  these  factors  fall  within  the  various  programs  of  the  health  care/promotion  functional  area 
(specifically  the  health  and  physical  readiness  program),  physical  fitness  efforts  might  also  be 
considered  under  the  welfare,  morale,  and  recreation  functional  area.  Alcohol  and  drug  education 
programs  fall  within  the  HPP/EAP  arena,  but  were  discussed  above  because  evaluation  efforts  in 
that  area  are  typically  independent  of  HPP  or  EAP  evaluations.  Because  HPPs  and  EAPs  do  not 
correspond  directly  to  Navy  programs,  the  methods  used  to  evaluate  these  private-sector  programs 
must  be  adapted  for  use  with  similar  military  programs.  Likewise,  one  cannot  assume  that  results 
are  directly  applicable  to  military  programs. 

Battle  (1988)  and  Yamatani  (1988)  provide  overviews  of  evaluation  methodology  in  this  area. 
Gibbs,  Mulvaney,  Henes,  and  Reed  (1985);  Hollander  and  Lengermann  (1988);  Kronenfeld, 
Jackson,  Davis,  and  Blair  (1988);  and  Bly,  Jones,  and  Richardson  (1986)  present  the  results  of 
evaluation  studies  of  HPPs/EAPs,  while  Flakenberg  (1987)  reviews  the  literature  evaluating  one 
aspect  of  HPPs,  and  Wheeler  (1988)  and  Smith,  Haight,  and  Everly  (1988)  evaluate  other  sub¬ 
elements  of  HPPs.  The  Employee  Assistance  Quarterly  is  a  relatively  new  journal  devoted  to  these 
issues.  Its  third  volume  (1988)  was  devoted  to  evaluation  issues  and  case  studies  in  this  area. 

Jerrell  and  Rightmyer  (1982)  reviewed  the  literature  available  at  that  time  on  the  effectiveness 
of  EAPs  (38  articles).  As  introductory  information,  they  provide  a  profile  of  the  characteristics  of 
programs  reviewed  and  the  typical  client.  In  addition,  they  review  the  many  ways  in  which  these 
programs  have  been  evaluated.  Various  program  evaluations  have  revealed  that  while  on-site 
accident  rates  are  similar  for  “troubled”  and  “non  troubled”  employees,  off-site  accident  rates  are 
higher  for  “troubled”  employees  (Brenner,  1967;  Trice  &  Roman,  1972);  EAPs  have  been 
associated  with  40  to  80  percent  reductions  in  on-  and  off-site  accidents  (Allander  &  Campbell, 
1975)  and  43  to  50  percent  reductions  in  absenteeism  (Allander  &  Campbell,  1975;  Pell  & 
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D’Alonzo,  1970);  and  treatment  costs  range  from  $840  to  $3,000  per  employee  per  year  (Allander 
&  Campbell,  1975;  Manello  &  Seamans,  1979). 


The  methodological  deficiencies,  lack  of  common  criterion  and  standard  measures,  and 
unstated  assumptions  about  cost  data  found  in  the  review  by  Jerrell  and  Rightmyer  limit  the  ability 
to  compare  results  from  study  to  study  or  to  combine  results  across  studies.  The  authors  suggest 
several  strategies  to  be  used  in  evaluating  EAPs  in  the  future. 

Hollander  and  Lengermann  (1988)  surveyed  Fortune  500  firms  to  determine  the  extent  and 
nature  of  worksite  HPPs.  As  part  of  the  survey,  they  also  assessed  the  extent  to  which  firms 
conducted  program  evaluations  or  cost-effectiveness  analysis  of  their  HPPs.  While  two  thirds  of 
responding  organizations  have  HPPs,  only  35  percent  reported  they  had  evaluated  their  programs 
and  only  16  percent  used  any  type  of  cost  analysis.  It  appears  that  many  organizations  implement 
HPPs  believing  they  will  have  positive  effects  without  determining  if  this  is  so.  At  a  minimum, 
organizations  should  determine  employees’  satisfaction  with  the  program.  Among  potential 
benefits  of  such  programs  cited  by  Hollander  and  Lengermann  are  reduced  health  care  and  health 
insurance  costs,  reduced  absenteeism,  reduced  on-the-job  accidents,  reduced  turnover,  increased 
productivity,  and  increased  employee  morale.  However,  adequate  research  is  not  yet  available  to 
support  these  claims. 

Battle  (1988)  examines  a  number  of  important  issues  to  be  considered  in  planning  evaluations 
of  EAPs  and  presents  a  model  for  use  in  developing  such  efforts.  Yamatani  (1988)  provides  a 
methodology  for  translating  the  benefits  and  costs  of  EAPs  into  dollar  figures  for  use  in  evaluating 
the  cost-effectiveness  of  such  programs  and  determining  break-even  points.  The  presentation  is 
non  technical  in  nature,  geared  to  the  EAP  practitioner. 


Gibbs,  Mulvaney,  Henes,  and  Reed  (1985)  describe  an  evaluation  of  the  first  5  years  of 
Blue  Cross  and  Blue  Shield  of  Indiana’s  HPP.  Short-  and  long-term  utilization  of  health  care 
services  are  evaluated  as  well  as  program  costs  relative  to  health  cost  reductions  for  participants. 
An  appropriate  comparison  group  of  non-participants  is  also  used  in  this  evaluation.  This  is  not  a 
cost-benefit  study  because  many  other  relevant  costs  and  benefits  (e.g,  those  related  to  absenteeism 
and  longevity)  are  not  considered.  The  authors  found  that  while  participants  incurred  higher  health 
care  costs  during  the  first  6  months  after  implementation,  participants  averaged  24  percent  lower 
health  care  costs  than  nonparticipants  over  5  years  of  program  operation.  Savings  in  5-year  health 
care  costs  exceed  program  costs  by  a  factor  of  1.45,  indicating  the  program  was  cost-effective  to 
operate. 

Kronenfeld,  Jackson,  Davis,  and  Blair  (1988)  used  questionnaire  methodology  on  health 
behaviors  and  health  attitudes  to  assess  the  effects  of  a  worksite  HPP  for  state  government 
employees  in  South  Carolina.  Because  no  records  were  kept  of  who  “joined”  the  program  and  who 
did  not,  all  results  reflect  the  impact  of  the  HPP  on  the  employee  cohort  as  a  whole.  No  attempt 
was  made  to  compare  behavioral  change  between  participating  and  non-participating  employees. 
While  this  study  includes  the  limitations  inherent  in  using  self-report  data  to  evaluate  such 
programs,  the  authors  did  employ  a  pre-post  longitudinal  design  to  assess  behavioral  changes  up 
to  1  year  after  program  implementation.  They  found  positive  changes  in  the  areas  of:  smoking,  seat 
belt  usage,  diet,  exercise,  and  alcohol  usage. 
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Bly,  Jones,  and  Richardson  (1986)  evaluated  Johnson  &  Johnson’s  HPP  by  comparing 
outcomes  at  independent  intervention  and  control  sites.  They  examined  changes  in  health 
behaviors,  health  care  costs,  and  hospital  admissions  over  a  5  year  period.  As  in  the  Kronenfeld  et 
al  (1988)  study,  these  authors  did  not  distinguish  between  employees  who  did  and  did  not 
participate  in  the  program  at  the  intervention  site.  Program  evaluation  compared  changes  between 
the  intervention  and  control  sites  as  a  whole.  Analysis  revealed  that  compared  to  control  sites, 
intervention  sites  showed  more  cessation  of  smoking,  more  weight  loss,  more  initiation  of  exercise 
programs,  and  less  increases  in  health  care  costs  and  utilization.  While  medical  costs  for 
participants  doubled  over  the  5-year  evaluation  period,  similar  costs  for  nonparticipants  increased 
four-fold.  For  inpatient  care  alone,  this  translates  to  an  annual  cost  savings  of  approximately  $34 
per  person. 

Smith,  Haight,  and  Everly  (1988)  evaluated  the  effects  of  a  corporate-sponsored  weight  loss 
program  (at  Kimberly-Clark).  They  evaluated  weight  reduction,  sick  leave  usage,  and  health  care 
costs  among  participants  and  a  matched  sample  of  non-participants  to  assess  the  effectiveness  of 
the  program.  Evaluation  data  were  collected  for  18  months  before  and  after  program 
implementation.  Direct  and  indirect,  fixed  and  variable  costs  of  the  program  were  identified.  Sick 
leave  reductions,  health  care  cost  savings,  and  program  fees  collected  from  participants  were  used 
to  measure  program  benefits.  The  authors  provide  a  clear  presentation  of  cost-effectiveness  and 
cost-benefit  calculations  for  this  program.  The  calculated  cost-benefit  ratios  and  cost-effectiveness 
measures  were  all  favorable.  The  most  conservative  calculations  showed  a  cost  benefit  ratio  of 
1.05/1  and  a  cost-effectiveness  ratio  of  $20  per  pound  loss.  The  authors  observe  that  the  cost- 
benefit  ratios  might  have  been  significantly  higher  had  less  conservative  assumptions  been  used 
and  had  other  potential  benefits  been  measured. 

In  reviewing  the  literature  on  one  aspect  of  HPPs— exercise-Falkenberg  (1987)  found  that  there 
is  relatively  little  evidence  that  employee  fitness  programs  lower  absenteeism  or  turnover. 
According  to  Falkenberg,  studies  in  this  area  rely  too  much  on  subjective  reports  of  change  (as 
some  of  the  previous  citations  indicate)  and  lack  adequate  controls.  One  study  that  surpassed  others 
in  objectivity  and  use  of  controls  and  might  be  used  as  an  example  is  reported  by  Cox,  Shepard, 
and  Corey  (1981)  and  Shepard,  Cox,  and  Corey  (1981).  This  study  included  the  use  of  a  control 
group  of  non-participants  to  evaluate  the  effects  of  the  exercise  program  on  turnover,  absenteeism, 
productivity,  and  job  satisfaction  up  to  6  months  after  introduction  of  the  program.  They  found  that 
participants  in  the  fitness  program  had  22  percent  less  absenteeism  and  1 3  percent  less  turnover 
than  nonparticipants.  The  authors  estimate  that  a  potential  for  a  1  percent  reduction  in  payroll  costs 
could  result  from  these  improvements.  No  difference  was  found  between  participants  and 
nonparticipants  in  productivity  or  job  satisfaction. 

Zimmerman,  Gerace,  Smith,  and  Benezra  (1988)  examined  the  effects  of  a  HPP  on  the  wives 
of  participating  employees.  Using  interview  data  from  a  small  sample  of  wives  of  participating  and 
non-participating  employees,  the  authors  compared  the  spouses’  knowledge  of  the  program  and 
health  habit  changes,  as  well  as  spouses’  perception  of  the  health  habit  changes  of  their  husbands. 
While  no  significant  changes  were  found  in  the  wives’  health  habits,  the  authors  did  find  the  wives 
were  more  aware  of  the  program  than  controls.  While  they  acknowledge  that  the  effort  was 
primarily  exploratory  in  nature,  their  work  does  suggest  an  additional  area  for  identifying  potential 
benefits  of  such  programs.  If  “spin-off’  benefits  can  be  identified  and  measured  among  family 
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members  of  participants  in  HPPs  or  EAPs,  the  cost-effectiveness  of  these  programs  will  be 
enhanced. 

Wheeler  (1988)  evaluated  the  effectiveness  of  a  community- wide  smoking  cessation  program 
immediately  after  completion  of  the  20-day  program  and  3  months  later.  Results  showed  that  16 
percent  had  stopped  smoking  by  the  end  of  the  program  and  12  percent  were  not  smoking  3  months 
later.  Cigarette  consumption  dropped  for  60  percent  of  participants.  While  no  comparison  or 
control  groups  were  used,  some  efforts  were  made  to  assess  the  fiscal  impact  of  behavioral 
changes.  Including  costs  for  health  care  for  smokers  and  their  employers,  and  costs  for  cigarettes, 
the  authors  translated  these  outcomes  into  an  annual  overall  cost  savings  of  $7,694,000.  Additional 
analyses  focused  on  identifying  successful  and  nonsuccessful  sub-groups. 

Spilman  (1988)  examined  participation  rates  in  a  corporate  HPP.  HPPs,  as  well  as  many  other 
quality  of  life  programs,  cannot  be  termed  successes  if  targeted  populations  do  not  participate. 
Participation  rates  for  men  and  women  were  compared  for  each  element  of  the  HPP  (e.g.,  stress 
management,  weight  reduction,  back  pain  management,  etc.).  This  methodology  can  be  used  to 
assess  use  of  a  wide  variety  of  quality  of  life  programs. 

At  this  time,  there  appears  to  be  some  evidence  of  the  effectiveness  of  HPPs  and  EAPs.  As 
Kozlowski  and  Oleksy  (1987)  point  out,  the  real  benefits  of  these  programs  might  not  become 
apparent  for  many  years  after  implementation.  The  ultimate  impact  will  be  known  only  when 
participants  reach  the  age  at  which  hypertension  and  other  negative  health  factors  take  their  toll. 
As  these  programs  become  institutionalized  in  corporate  America  and  long-term  evaluation  efforts 
are  conducted,  additional  evidence  of  their  success  or  failure  should  become  available. 

Alcohol  and  Drug  Abuse  Control 

The  greatest  number  of  evaluation  efforts  within  the  Navy’s  health  care/prevention  functional 
area  were  found  in  the  alcohol  and  drug  education  and  treatment  area.  Many  of  these  studies  focus 
on  the  evaluation  of  specific  programs  or  innovative  treatment  methods  or  provide  evaluation 
guidelines  or  methods  (e.g.,  Barnes,  1984;  Kim,  Hoffman,  Pike,  &  Gibson,  1984;  Mitchell,  Hu, 
McDonnell,  &  Swisher,  1984;  Moskowitz,  Malvin,  Schaeffer,  &  Schaps,  1984;  Retka,  1977; 
Werch,  &  Damron,  1985).  A  portion  of  these  studies  report  the  results  of  evaluation  efforts 
focussed  on  DoD  or  Navy  programs  (e.g.,  Booz,  Allen,  &  Hamilton,  1981;  Borthwick,  1977;  Hoyt, 
1974;  Kolb  &  Gunderson,  1975;  Kolb,  Gunderson,  &  Cohen,  1978;  Resource  Consultants,  Inc., 
1980;  Systems  Development  Corporation,  1975).  The  articles  and  documents  cited  here  do  not 
provide  an  exhaustive  list  of  evaluation  efforts  in  this  area.  The  Journal  of  Drug  Education  and  the 
Quarterly  Journal  of  Studies  on  Alcohol  routinely  publish  such  work  and  are  good  sources  of 
illustrative  evaluation  efforts  in  this  area. 

Cairo  (1983)  estimates  that  the  costs  to  industry  related  to  alcoholism  reach  $12  billion 
annually,  taking  into  account  the  effects  of  absenteeism,  tardiness,  accidents,  turnover,  substandard 
job  performance,  and  the  costs  of  treatment.  The  costs  associated  with  drug  abuse,  though  likely 
not  as  large,  are  substantial  as  well.  Efforts  to  prevent  or  treat  alcoholism  and  drug  abuse  can  be 
assessed  in  part  by  their  effects  on  the  factors  listed  above.  Cairo  estimates  that  between  1,200  and 
2,400  employer-sponsored  treatment  programs  exist  for  alcohol  abuse  alone. 
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In  a  review  of  these  programs,  Follman  (1978)  found  recovery  rates  of  60  to  80  percent  were 
common.  The  most  common  criterion  for  success  is  return  to  work  with  no  recurrence  of  the 
problem,  which  prompted  referral  to  the  treatment  program.  However,  the  time  period  over  which 
a  treated  employee’s  performance  is  examined  varies  widely.  Typically,  success  is  judged  1  to  2 
years  after  treatment.  Discrepancies  in  this  measure  limit  the  ability  to  compare  studies. 

Borthwick  (1977)  provides  the  best  example  of  evaluation  research  of  Navy  and  Marine  Corps 
programs  in  this  area.  Borthwick  summarizes  the  results  of  a  cost-benefit  analysis  of  the  Navy’s 
Alcoholism  Prevention  Program  (NAPP).  In  addition  to  assessing  treatment  effectiveness  of 
NAPP,  the  study  also  examined  the  effects  of  Navy  resident  treatment  programs  on  replacement, 
hospitalization,  accidents,  jurisprudence,  and  productivity  aspects  of  military  operations.  These 
aspects  are  included  in  the  cost-effectiveness  analysis.  Borthwick  found  that  discharging  identified 
alcoholics  and  replacing  them  with  new  personnel  is  2.2  times  more  costly  than  the  existing 
rehabilitation  program.  This  rate  rises  to  five  times  more  costly  for  more  senior  personnel  (ages  26 
and  older).  Treatment  effectiveness  of  the  existing  program  (based  on  a  2-year  post-treatment 
evaluation)  is  83  percent  for  the  26  and  older  group  and  44  percent  for  those  25  and  younger. 
Successful  rehabilitation  of  Navy  and  Marine  Corps  alcoholics  reduces  their  sick  day  rate  from  9 
days  per  person  per  year  to  the  service  average  of  2.7  days  per  person  per  year.  Effective 
rehabilitation  translates  to  a  cost  avoidance  of  $7.8  million  in  reduced  demand  for  health  care 
services. 

Systems  Development  Corporation  (1975)  presents  the  results  of  an  evaluation  of  DoD 
Alcohol  Abuse  Control  Programs  (AACP).  Thirty  military  sites  in  the  U.S  and  overseas  were 
visited  to  evaluate  the  four  components  of  the  DoD  AACP:  education/prevention  programs, 
treatment/rehabilitation  programs,  identification  programs,  and  program  administration. 
Evaluation  was  at  the  DoD  level,  rather  than  by  service  or  site.  In  addition  to  describing  the  types 
of  programs  in  each  component,  the  measures  of  effectiveness  used  for  each  are  presented.  Some 
measures  are  archival  (collected  from  records  maintained  by  each  site).  However,  the  majority  are 
subjective  (e.g.,  client  and  administrator  ratings  of  program  effectiveness  and  recidivism).  Results 
of  the  study  include  in-house  evaluation  of  education/prevention  programs  is  weak,  Alcoholics 
Anonymous  meetings  and  peer  group  therapy  were  the  most  effective  treatment  modalities,  use  of 
Antabuse  was  the  most  ineffective  method,  funding  inadequacy  was  seen  as  a  consistent  weakness 
in  the  treatment  program,  and  no  clear  difference  in  effectiveness  of  centralized  or  local  treatment 
facilities  exists. 

Hoyt  (1974)  evaluated  the  effectiveness  of  the  Navy’s  Drug  Education  Program  by 
administering  questionnaires  to  a  sample  of  Navy  commands  and  interviewing  program 
administrators.  Questionnaire  items  covered  four  major  areas:  drug  knowledge,  attitudes  toward 
drugs  and  drug  users,  use  of  Drug  Education  Program  resources,  and  drug  utilization.  While  the 
use  of  self-report  data  to  assess  the  effects  of  drug  education  on  drug  use  is  less  than  fully 
dependable,  the  remaining  evaluation  areas  are  covered  adequately.  Areas  for  improvement  of  the 
program  were  also  identified.  The  author  concluded  that  exposure  to  drug  education  resources 
increases  drug  knowledge,  but  has  no  apparent  effect  on  drug  use.  However,  the  more  important 
finding  was  that  tne  drug  education  program  had  received  little  command  support  and,  thus,  had 
not  been  utilized  adequately. 
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Resource  Consultants,  Inc.  (1980)  compared  the  size,  effectiveness,  and  efficiency  of  the 
Navy’s  alcohol  and  drug  abuse  programs  to  those  of  the  Army  and  Air  Force.  Data  for  each  service 
were  obtained  from  historic  records— program  justification  data  provided  by  each  service  each  year 
to  the  Assistant  Secretary  of  Defense  (Health  Affairs).  The  report  compares  the  services’ 
performance  in  the  areas  of  identification  and  treatment  and  calculates  cost  per  client  for  each 
service.  Problems  with  using  archival  data  for  evaluation  were  mentioned  (e.g.,  inconsistent 
reporting  of  data,  incomparable  treatment  effectiveness  periods  among  the  services,  etc.). 

Their  analysis  showed  that  Navy  was  more  effective  in  identifying  individuals  with  alcohol  and 
drug  problems  (due  to  its  Navy  Safety  Action  Program)  than  the  other  services.  Navy  had  3.4 
percent  of  active  duty  personnel  enter  alcohol  treatment  programs,  while  the  Army  had  only  1.8 
percent  and  the  Air  Force  1  percent.  Similar  figures  were  reported  for  individuals  entering  drug 
treatment  programs.  The  effectiveness  rate  for  Navy’s  alcohol  rehabilitation  programs  is  71 
percent  (compared  to  65  percent  for  Army  and  Air  Force).  Comparing  the  efficiency  of  these 
programs  (calculated  as  dollars  spent  per  persons  successfully  treated  in  1978),  Navy  is  most 
efficient.  Navy  spent  $848  per  rehabilitee,  while  the  Army  spent  $1,124  and  the  Air  Force  spent 
$2,896.  At  the  time  the  analysis  was  conducted,  Navy  and  Army  effectiveness  rates  for  drug 
treatment  programs  were  not  known.  Air  Force  spent  about  $750  per  successful  rehabilitee  and 
reported  an  effectiveness  rate  of  54  percent.  Assuming  a  comparable  effectiveness  rate  for  Navy 
drug  treatment  programs,  their  cost  per  rehabilitee  would  have  been  $2,512. 

Booz,  Allen,  and  Hamilton  (1981)  compared  the  Navy’s  drug  abuse  control  program  to  that  of 
the  Air  Force.  They  examined  the  organizational  responsibility  for  the  program,  activities 
comprising  the  program,  management  of  the  program,  and  resources  committed  to  the  program.  In 
doing  so,  they  identified  shortcomings  in  the  Navy’s  system.  While  this  is  not  a  true  program 
evaluation  study  (no  quantitative  evaluation  was  conducted),  such  comparative  analyses  can 
provide  information  about  strengths  and  weakness  of  a  program  and  identify  improvements  that 
can  be  made. 

Kolb  and  Gunderson  (1975)  and  Kolb,  Gunderson,  and  Cohen  (1978)  report  on  post-treatment 
outcomes  for  Navy  men  treated  for  alcoholism  and  attempt  to  identify  differences  in  effectiveness 
among  patient  sub-groups.  Further,  they  cite  the  need  for  research  to  identify  the  successful 
elements  of  alternative  treatment  programs.  Their  work  underscores  the  fact  that  overall  treatment 
effectiveness  measures  may  be  misleading.  More  useful  information  is  provided  when 
effectiveness  measures  are  calculated  for  significant  sub-groups  of  the  client  population.  Likewise, 
sub-elements  of  treatment  programs  should  ideally  be  assessed  independently.  Their  analyses 
revealed  that  effectiveness  rates  ranged  from  54  to  91  percent  depending  on  type  of  treatment  and 
age  of  participant.  Higher  effectiveness  rates  were  found  for  older  men  (89%  vs.  59%  for  younger 
men)  and  for  individuals  treated  in  dry  docks  (63%  for  younger  men  and  91%  for  older  men). 

These  studies  of  military  alcohol  and  drug  treatment  programs  reveal  moderate  effectiveness 
rates  of  the  Navy’s  programs.  From  this  limited  review,  it  appears  the  Navy’s  education  and 
awareness  programs  are  far  less  effective  than  its  identification  and  treatment  efforts. 

Turning  to  the  non-DoD  efforts,  Retka  (1977)  provides  an  overview  of  a  methodology  for 
calculating  cost  effectiveness  of  drug  abuse  prevention  programs.  This  report  includes  a  discussion 
of  the  costs  of  dysfunctional  drug  use  and  the  costs  associated  with  four  model  programs  used  by 
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the  National  Institute  on  Drug  Abuse.  Retka  concluded  that  all  four  model  programs  can  be  cost 
effective  if  a  10  percent  level  of  effectiveness  can  be  achieved. 

Mitchel,  Hu,  McDonnell,  and  Swisher  (1984)  present  the  results  of  a  cost-effectiveness 
analysis  using  a  quasi-experimental  approach  to  evaluate  four  variations  in  an  educational  drug 
abuse  prevention  program  for  high  school  students.  They  found  that  one  variation  (imbedding 
alcohol  and  drug  information  in  a  religious  studies  class)  was  effective  in  changing  behaviors  at  a 
cost  of  $68  per  student.  None  of  the  other  alternatives  evaluated  were  effective. 

Leveson  (1973)  presents  a  model  for  conducting  cost-benefit  analysis  that  takes  into  account 
both  characteristics  of  the  client  population  and  variations  in  treatment  programs.  It  is  used  to 
evaluate  narcotics  addiction  treatment  programs.  His  analysis  revealed  that  a  Methadone  treatment 
program  was  more  effective  (70%  success  rate  after  36  months)  than  the  alternative  methods  being 
used  in  New  York  State  (incarceration  and  rehabilitation  services  resulting  in  44%  success  rate 
after  21  months). 

This  group  of  studies  provides  information  that  can  be  useful  in  developing  a  cost-effectiveness 
or  cost-benefit  analysis  of  alcohol  and  drug  programs.  Some  of  the  points  made  in  these  papers  and 
the  methodologies  they  suggest  can  be  applied  to  similar  evaluations  in  other  quality  of  life 
functional  areas.  The  following  group  of  studies  assess  treatment  effectiveness  or  describe 
evaluation  methodologies. 

Miller  (1988)  repons  on  the  evaluation  of  a  chemical  dependence  treatment  program  that  used 
three  types  of  outcome  measures  in  addition  to  the  traditional  measure  that  assesses  substance  use/ 
abuse  after  treatment.  A  treatment  program,  shown  to  be  successful  in  reducing  drug  usage,  was 
also  found  to  be  successful  in  reducing  clients’  presenting  symptoms,  changing  clients’  attitudes, 
and  improving  clients’  quality  of  life.  This  study  shows  the  usefulness  of  self-report  data  in 
supplementing  program  effectiveness  information. 

Werch  and  Damron  (1985)  present  an  experimental  study  of  the  effectiveness  of  an  alcohol 
control  program  designed  to  reduce  drinking  and  driving.  While  much  of  them  evaluation  data  is 
subjective,  the  design  of  the  study  is  sound.  Unlike  previous  evaluations  of  the  same  alcohol 
control  program,  the  current  study  found  no  significant  effects  of  program  participation.  The 
authors  speculate  that  this  may  be  due  to  the  non-problem  drinker  population  treated  in  the  present 
study. 

Likewise,  Moskowitz,  Malvin,  Schaeffer,  and  Schaps  (1984)  employed  an  experimental  design 
to  assess  the  effectiveness  of  a  drug  education  course  for  seventh  graders.  They  used  both 
subjective  data  (e.g.,  intention  to  use  drugs)  and  a  drug  knowledge  test  to  evaluate  the  program. 
Results  indicated  the  program  had  no  significant  effect  on  girls  and  only  a  few  significant  effects 
on  boys. 

Kim,  Hoffman,  Pike,  and  Gibson  (1984)  present  a  theoretically-based  outcome  evaluation 
instrument  that  can  be  used  in  assessing  the  effectiveness  of  alcohol  education,  prevention,  and 
intervention  programs.  Barnes  (1984)  reviews  previous  efforts  in  evaluating  alcohol  and  drug 
education  programs  and  identifies  their  deficiencies.  This  group  of  studies  provides  examples  of 
experimental  evaluation  studies,  suggests  a  number  of  outcome  measures  that  can  be  considered  in 
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evaluating  the  effectiveness  of  alcohol  and  drug  programs,  and  identifies  problems  to  be  avoided. 
Further,  they  reveal  the  extent  to  which  different  treatment  methods  can  have  different 
effectiveness  rates.  Making  global  conclusions  about  the  effectiveness  of  alcohol  or  dmg  treatment 
programs  is  unwarranted.  Each  program  must  be  evaluated  independently. 

Housing 

This  functional  area  includes  two  programs:  bachelor  quarters  and  military  family  housing. 
Private  sector  organizations  rarely  provide  housing  to  their  employees.  Thus,  few  evaluation 
studies  were  found  in  this  area. 

Bachelor  Housing 

Resource  Consultants,  Inc.  (1980)  reported  the  results  of  a  world-wide  bachelor  housing  survey 
conducted  by  the  Navy  in  1979.  This  survey  found  that  the  Navy  is  66,239  berths  short  in  bachelor 
quarters  (958,012  enlisted  berths  and  8,227  officer  berths)-a  shortfall  of  about  46  percent  of  the 
need.  Typically,  this  deficiency  is  met  in  the  form  of  Basic  Allowance  Quarters  (BAQ) 
supplements  in  lieu  of  quarters.  The  authors  identify  ambitious  programs  designed  to  meet  this 
shortfall,  but  indicate  that  proposed  reductions  may  jeopardize  the  effort. 

Family  Housing 

Resource  Consultants,  Inc.  (1980)  found  that  between  1976  and  1979  approximately  15  percent 
of  Navy  permanent  change  of  station  (PCS)  families  moving  into  military  housing  were  not 
suitably  housed.  The  percentages  for  the  other  services  were  consistently  lower.  Looking  at  private 
housing,  they  found  that  while  the  proportion  of  Navy  families  suitably  housed  improved  from 
1976  to  1979,  the  other  services  nonetheless  had  better  suitable  housing  rates.  In  addition,  the 
authors  found  that  the  Navy  spends  less  per  capita  on  operating  costs  for  family  housing  (e.g., 
utilities,  repairs,  and  maintenance),  than  the  other  services  do.  The  total  expenditures  for  operating 
costs  and  investments  (e.g.,  new  construction,  renovation,  and  modernization)  in  the  Navy  are  far 
less  than  in  the  other  services.  The  Navy  spends  $624  per  person;  the  Army  spends  $796;  and  the 
Air  Force  spends  $932.  Navy  does  exceed  Army  in  providing  funds  for  improvement  in  existing 
family  housing  units.  The  authors  speculate  that  this  consistent  lower  level  of  support  for  family 
housing  may  ultimately  have  a  negative  impact  on  retention. 

A  1977  General  Accounting  Office  (GAO)  study  examined  DoD’s  self-help  maintenance 
program  for  occupants  of  military  family  housing.  This  voluntary  program  is  designed  to 
encourage  occupants  to  perform  minor  maintenance  tasks  on  their  homes  and  develop  owner-like 
responsibility.  While  the  services  believe  their  programs  are  effective,  they  have  no  empirical 
evidence  to  determine  if  self-help  programs  are  cost-effective.  GAO  found  that  these  programs 
have  weak  controls  over  funds,  inventory,  and  materials  and  have  no  acceptable  cost  accounting 
procedures.  Further,  the  program  is  voluntary  and  there  is  no  incentive  for  occupants  to  perform 
self-help  maintenance  since  other  means  are  available  to  accomplish  such  tasks. 

In  the  same  study,  GAO  also  examined  the  effectiveness  of  the  services  in  charging  and 
collecting  damages  from  tenants.  They  found  that  all  services  lack  explicit  criteria  for 
distinguishing  damages  from  normal  wear-and-tear  and  lack  adequate  procedures  for  identifying 
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damages.  Consequently,  damages  are  often  treated  as  normal  maintenance  tasks  and  repaired  at  the 
services’  cost.  Based  on  a  survey  of  1 1  bases  (comprising  about  8%  of  DoD’s  housing  units),  GAO 
found  over  $1  million  in  repair  costs  during  a  1  year  period  that  resulted  from  occupant  damages 
but  were  not  collected  from  the  tenants  (e.g.,  broken  doors,  holes  in  walls  and  ceilings,  missing 
towel  racks  and  curtain  rods,  and  tom  screens). 

Personnel  Policies 

Evaluating  the  effects  of  personnel  policies  such  as  sea/shore  rotation  policy,  equal 
opportunity,  and  work  environment  on  any  tangible  outcome  such  as  retention  or  readiness  is  very 
difficult.  As  in  many  quality  of  life  areas,  individuals  and  organizations  often  have  strong  beliefs 
that  a  particular  change  to  any  of  these  policies  will  have  or  have  had  a  positive  effect  on  the  Navy 
and  its  personnel;  these  claims  are  rarely  evaluated.  Few  studies  in  this  area  were  found. 

Resource  Consultants,  Inc.  (1980)  compared  the  average  Navy  work  week  to  the  Navy  goals 
and  to  the  average  civilian  work  week.  They  found  that  the  typical  American  worker  is  at  the  job 
1,888  hours  per  year  while  the  average  sailor  aboard  ship  is  on-board  2,828  hours  per  year  if  in  port 
the  entire  year.  This  includes  22.9  weekend  days.  The  Navy  standard  work  week  calls  for  59.25 
hours  per  week  aboard  ship.  The  authors  found  that  in  surface  combatants  and  submarines  of  the 
Pacific  Fleet,  sailors  exceed  this  by  20  percent.  This  translates  to  a  64  percent  increase  over  the 
typical  civilian  work  week.  As  of  the  time  of  their  review,  further  efforts  were  needed  to  reach  the 
Navy  work  week  goal. 

Resource  Consultants,  Inc.  (1980)  also  examined  changes  in  sea-shore  rotation  between  1974 
and  1979.  They  found  a  marked  improvement  between  these  two  times.  While  in  the  past,  junior 
enlisted  personnel  could  expect  10  years  of  sea  duty  for  every  3  years  of  shore  duty,  in  1979  they 
could  expect  7.7  years  of  sea  duty  for  3  years  of  shore  duty.  Similar  improvements  were  found  for 
senior  enlisted  personnel.  No  analysis  of  the  impact  of  this  change  on  retention  or  performance  was 
made. 

Education/T raining  Programs 

The  Navy  provides  many  off-duty  voluntary  educational  opportunities  to  its  members  through 
the  Navy  Campus  for  Achievement  (NCFA).  Many  non-military  employers  provide  similar 
benefits  to  their  employees.  Githens  and  Wilcove  (1977)  reviewed  the  literature  related  to 
employer-sponsored  off-duty  education  and  found  “no  empirical  evidence  that  clearly  linked  off- 
duty  education  to  recruiting,  performance,  or  retention  “(p.  5).  In  a  similar  survey  (Milwaukee 
Personnel  Department  Training  Unit,  1977),  efforts  were  made  to  identify  studies  of  the 
quantitative  cost-benefits  of  tuition  reimbursement  programs.  The  authors  concluded  that  “such  an 
analysis  does  not  exist  in  either  the  public  or  the  private  sector”  (p.  170).  Despite  the  lack  of 
adequate  evaluation  in  this  area,  Githens  and  Wilcove  identified  a  survey  of  279  companies  that 
showed  that  81  percent  provide  monetary  assistance  to  employees  for  education  (Administrative 
Management  Society,  1976). 


21 


In  another  survey  of  283  firms,  Watson  and  Grzybowski  (1975)  found  that  over  90  percent  have 
some  form  of  tuition  plan.  Almost  all  firms  participating  in  the  survey  reported  they  had  benefited 
from  their  plans  and  54  percent  reported  that  employees  using  the  tuition  support  plans  had 
advanced  in  the  organization  faster  than  others.  Organizations  also  reported  increased  quality  of 
work  and  improved  organizational  climate.  However,  these  conclusions  were  based  on  self-report 
data  from  firms  participating  in  the  survey  rather  than  from  any  empirical  evaluation.  Githens  and 
Wilcove  (1977)  point  out  that  since  the  majority  of  private  sector  corporations  provide  educational 
assistance  of  some  sort,  the  Navy  must  do  likewise  to  maintain  a  competitive  recruiting  position, 
despite  the  lack  of  any  proven  cost-effectiveness  data. 

Githens  and  Wilcove  (1977)  analyzed  the  effects  of  the  Navy’s  off-duty  educational  programs 
on  recruiting,  performance,  and  retention.  Based  on  a  survey  of  recent  recruits,  they  found  that  45 
percent  indicated  that  they  joined  the  Navy  primarily  for  education  and  training  benefits  under  the 
Navy  Campus  for  Achievement  program.  Respondents  also  indicated  that  the  opportunity  to 
continue  their  education  was  a  fairly  strong  influence  on  their  decision  to  enlist.  From  these  results, 
the  authors  concluded  that  the  NCFA  appears  to  provide  a  positive  recruiting  incentive. 

The  Program  for  Afloat  College  Education  (PACE)  was  evaluated  by  comparing  responses  to 
a  Human  Resource  Management  (HRM)  Survey  between  high-PACE-participation  ships  and  a 
matched  group  of  non-PACE-participation  ships.  Githens  and  Wilcove  (1977)  found  that 
personnel  aboard  high-PACE-participation  ships  perceived  organizational  conditions  significantly 
more  favorably.  The  authors  also  compared  the  promotion  and  retention  rates  between  a  matched 
sample  of  participants  and  nonparticipants  in  the  PACE,  Tuition  Assistance,  and  Contract  for 
Degree  programs.  They  found  that  participants  in  the  PACE  and  Tuition  Assistance  programs  were 
not  as  likely  to  reenlist  as  nonparticipants,  but  that  participants  in  the  Tuition  Assistance  and 
Contract  for  Degree  programs  were  more  likely  to  be  promoted  than  nonparticipants.  Githens  and 
Wilcove  speculate  that  personnel  who  are  planning  to  leave  the  service  participate  in  these 
programs  to  better  prepare  themselves  for  civilian  jobs. 

The  current  job  performance  levels  of  NCFA  participants  and  nonparticipants  were  also 
compared.  No  differences  were  found  between  the  two  groups  on  performance  indices  such  as 
military  performance  marks,  disciplinary  actions,  or  commendations.  However,  the  authors  note 
that  NCFA  participants  have  higher  aptitude  scores  than  nonparticipants  and  may  perform  better 
overall. 

Based  on  their  many  varied  analyses,  Githens  and  Wilcove  conclude  that  off-duty  educational 
programs  should  be  continued.  They  are  viewed  very  positively  by  civilian  industry,  recruits,  Navy 
wives,  operational  commands,  and  active  duty  personnel.  Their  positive  links  to  promotion  and 
perceptions  of  organizational  effectiveness  are  important. 

Resource  Consultants,  Inc.  (1980)  compared  Navy’s  off-duty  education  programs  to  those  of 
the  Army  and  Air  Force.  They  reported  that  the  PACE  program  selectively  services  only  6  percent 
of  the  personnel  afloat  and  that,  because  they  are  assigned  sea  duty,  few  junior  personnel 
participate.  Further,  Navy  spent  only  $25  per  capita  toward  these  programs,  while  the  Army  spent 
$  1 1 9  and  the  Air  Force  spent  $6 1  per  capita.  Finally,  the  ratio  of  off-duty  education  staff  per  10,000 
service  members  was  4  for  Navy  and  16  for  both  Army  and  Air  Force.  While  Githens  and  Wilcove 
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concluded  the  Navy’s  education  program  was  effective,  Resource  Consultants,  Inc.  conclude  that 
the  Navy  is  not  keeping  parity  with  the  other  services  in  this  area. 

Morale,  Welfare,  and  Recreation 

The  programs  provided  for  service  members  within  this  area  are  largely  unique  to  the  military 
services.  With  the  exception  of  child  care  services,  few  private  sector  organizations  provide  similar 
services  to  their  employees.  Furthermore,  those  somewhat  comparable  services  (e.g.,  employer- 
subsidized  cafeterias)  are  rarely  evaluated  independently.  Thus,  most  of  the  information  in  this 
section  pertains  entirely  to  military  services.  These  programs  are  designed  to  “maintain  high  levels 
of  espirit-de-corps,  job  proficiency,  military  effectiveness,  educational  attainment,  and  physical 
well-being”  (Resource  Consultants,  Inc.,  1980,  p.  A-l).  Few  studies  have  examined  their 
effectiveness  in  any  of  these  areas. 

Recreation/Clubs 

Resource  Consultants,  Inc.  (1980)  compared  the  morale,  recreation,  and  welfare  (MWR) 
programs  provided  by  each  of  the  armed  services.  They  examined  end-of-year  reports  submitted 
by  each  service  to  DoD.  Looking  at  overall  per  capita  expenditures  on  MWR  programs,  they  found 
that  Navy  spent  less  than  the  other  services  in  each  year  evaluated  (1976  to  1978).  In  1978,  the  Air 
Force  out  spent  Navy  by  15.4  percent.  (Note  that  active  duty  end-strength,  on  which  these 
calculations  were  based,  was  reduced  for  Navy  to  reflect  the  personnel  at  sea  and  physically 
removed  from  MWR  services.) 

Looking  at  the  military  welfare  and  recreation  program  (e.g,  pools,  bowling,  movies,  sports 
activities,  etc.),  they  found  a  difference  between  the  services.  The  difference  indicates  that  either 
the  Navy  fees  to  sailors  for  these  services  are  higher  than  those  in  the  other  services,  or  that  a 
greater  number  of  Navy  personnel  use  these  facilities,  thereby  yielding  greater  income. 

In  the  area  of  military  clubs,  their  comparison  showed  that  the  Air  Force  out  spent  the  Navy  by 
about  $100  per  capita  in  the  3  years  examined.  All  services  operated  at  a  net  loss  when  income  from 
dues  and  package  stores  was  subtracted.  The  Navy’s  loss  was  less  than  that  of  the  other  services. 

To  achieve  economic  parity  with  the  other  services  in  the  MWR  area,  as  a  whole.  Resource 
Consultants,  Inc.  (1970)  report  Navy  would  have  to  increase  support  for  these  programs  by 
somewhere  between  $25  million  and  $108  million  annually. 

A  1979  GAO  (1979a)  review  of  military  club  usage  indicated  that  Navy  enlisted  personnel  use 
their  clubs  less  than  their  counterparts  in  the  Army,  but  more  than  enlisted  personnel  in  the  Air 
Force.  The  Navy  officers  used  their  clubs  less  than  the  Army  and  Air  Force.  Most  often  given 
reasons  for  not  using  military  clubs  (across  all  services,  including  enlisted  personnel  and  officers) 
were:  atmosphere,  entertainment,  quality  of  services,  and  hours  of  operation.  Most  commonly 
given  reasons  for  joining  or  using  military  clubs  were  drink  prices  and  location.  For  the  most  part, 
different  reasons  were  given  by  enlisted  personnel  and  officers  and  different  reasons  were  given 
by  service  members  from  the  three  services. 
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Lalchandadi,  Morey,  deAndrade,  Humphreys,  and  Snyder  (1975)  used  an  innovative  technique 
to  assess  the  cost-effectiveness  of  the  Navy’s  recreation  program.  They  examined  the  potential 
impact  of  eliminating  recreation  services  in  terms  of  reduced  operating  costs  and  increased 
recruiting  and  training  costs  required  to  compensate  for  the  increased  turnover  that  would  result. 
Given  that  the  Navy  spent  $100  million  on  these  programs  in  1974,  the  potential  for  savings  by 
eliminating  undesired  services  is  substantial.  The  authors  administered  a  survey  to  a  broad  cross- 
section  of  eligible  users  (N  =  1 1,229)  to  determine  the  extent  of  usage  of  recreation  services.  In  this 
area,  they  found  that  bowling,  movies,  outdoor  recreation,  informal  sports,  and  other  recreation 
services  were  most  used.  Boating/sailing  and  golf  facilities  were  least  used.  Of  responding 
personnel,  13  percent  were  characterize  i  as  high  recreation  services  users  and  35  percent  were 
characterized  as  extremely  low  users. 

To  capture  the  value  of  recreation  services,  survey  respondents  indicated  that  an  increase  in 
salary  would  be  necessary  to  prevent  their  resignation  if  each  component  of  recreation  services  was 
eliminated.  In  other  words,  individuals  estimated  the  size  of  the  monthly  pay  increase  that  would 
be  required  to  make  the  person  indifferent  to  loss  of  the  service.  To  assess  the  perceived  value  of 
these  services  to  eligible  users,  the  survey  also  asked  respondents  to  estimate  the  dollar  savings 
each  component  of  recreation  services  is  worth  to  them  using  frequency  of  use  and  the  existence 
and  cost  of  comparable  off-base  facilities  as  inputs.  In  contrast  to  the  use  rates  reported  above, 
these  responses  indicated  that  the  auto  hobby  shop,  recreation  equipment  checkout,  and  bowling 
were  the  most  valued  by  the  users,  while  golf  facilities  and  organized  sports  were  least  valued.  For 
example,  respondents  indicated  they  would  require  $87.20  per  person  per  year  in  compensation  for 
eliminating  the  auto  hobby  shop,  but  only  $44.80  per  year  for  eliminating  the  golf  facilities.  By 
using  the  ratio  of  benefits  (based  on  both  value  calculations  above)  to  net  expenses  for  each 
component  of  recreation  services,  it  was  found  that  all  components  are  cost-effective. 

Finally,  econometric  models  used  these  data  to  estimate  the  impact  on  retention,  changes  in 
recruiting  and  training  costs  that  would  result  from  eliminating  recreation  services  without 
providing  alternative  compensation.  In  addition,  optimization  models  were  used  to  recommend 
resource  allocation  across  recreation  service  components. 

Child  Care 

Few  sound,  objective  studies  of  the  effectiveness  or  cost-benefits  of  employer-sponsored  child 
care  programs  are  available.  In  fact  the  Department  of  Labor  (1988)  reported  that  employers’ 
efforts  to  address  employees’  child  care  needs  are  occurring  prior  to  any  concrete  business  or 
industry  research  showing  a  quantifiable  positive  outcome  of  such  support  While  the  popular  press 
and  some  of  the  personnel  journals  routinely  publish  articles  touting  the  benefits  of  employer- 
sponsored  child  care,  little  empirical  evidence  is  available.  Such  articles  claim  that  employers  can 
reap  benefits  including  improved  ability  to  attract  and  retain  employees,  decreased  sick  leave,  and 
improved  employee  satisfaction.  As  with  HPPs  and  EAPs,  people  believe  employer-sponsored 
child  care  programs  are  “good”  things  to  do,  communicating  to  employees  that  management  cares 
about  them.  Tn  addition,  they  keep  corporations  competitive  in  recruiting  new  employees.  Until 
appropriate  evaluations  of  the  effects  of  employer-sponsored  child  care  programs  are  conducted, 
such  claims  cannot  be  supported. 
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Miller  (1984)  reviewed  the  literature  on  these  programs  and  concluded  that  “assertions  that 
employer-sponsored  child  care  reduces  workers’  absenteeism  or  tardiness,  or  that  it  increases 
workers’  productivity  or  job  satisfaction  are  not  supported  by  credible  research”  (p.  277).  Little 
solid  evidence  based  on  sound  research  methodology  has  been  offered  in  support  of  these 
programs.  Miller  cites  one  study  (Milkovich  &  Gomez,  1976)  that  made  an  attempt  to  compare  the 
work  behaviors  of  child  care  users  to  comparable  non- users. 

Milkovich  and  Gomez  (1976)  reported  on  the  North  Side  Child  Development  Center 
(Minneapolis)  in  which  Control  Data  Corporation  participated.  Work  behaviors  of  30  participating 
parents,  30  non-participating  parents,  and  30  others  were  compared.  Turnover  rates  were 
significantly  different  among  the  three  groups  (1.8%  for  participants,  6.3%  for  non-participant 
parents,  and  5.5%  for  others).  Likewise,  absenteeism  rates  were  significantly  different  (4.4%  for 
users,  6.0%  for  other  parents,  and  5.0%  for  others).  No  difference  was  found  in  the  job  performance 
rate.  Miller  acknowledged  that  this  study  made  an  excellent  attempt  to  introduce  control  in 
studying  this  issue  by  using  comparison  groups,  but  he  also  criticized  the  study  because  important 
information  about  the  comparison  groups  was  not  reported.  The  reasons  for  participating  in  the 
child  care  program  or  not  participating  were  not  given.  In  addition,  no  attempt  was  made  to 
measure  attendance  and  tardiness  before  enrolling  in  the  program.  Initial  differences  between  the 
groups  might  have  accounted  for  the  reported  findings. 

Between  1978  and  1985,  Department  of  Labor’s  (1988)  review  of  employer-assisted  child  care 
found  that  industry  participation  in  child-care  assistance  rose  more  than  2000  percent.  In  addition, 
labor  unions  are  becoming  involved  in  establishing  child  care  programs  for  members.  Nonetheless, 
they  found  that  only  11  percent  of  the  nation’s  worksites  (with  10  or  more  employees)  provide 
direct  child  care  help  for  their  employees.  Direct  help  includes  operating  on-site  or  nearby  child 
care  centers,  providing  financial  assistance  for  child  care,  and  providing  other  child  care  benefits 
such  as  referral  or  counseling  programs.  The  findings  of  the  Department  of  Labor’s  review  include 
quantified  outcomes  and  anecdotal  evidence  in  the  following  areas: 

1 .  Recruitment.  A  photography  company  reported  an  annual  savings  of  $30K  in  recruitment 
costs  as  a  result  of  offering  child  care  services  to  its  employees.  In  a  labor  market  with  only  2  to  3 
percent  unemployment,  they  reported  3,500  walk-in  job  applicants  in  one  year. 

2.  Retention.  A  Sioux  Falls  hospital  reported  that  many  parents  won’t  consider  working 
elsewhere  because  of  their  on-site  child  care  center.  They  believe  the  center  allows  them  to  keep 
qualified  medical  workers  who  are  in  short  supply  in  the  local  area. 

3.  Productivity.  A  hosiery  mill  reported  saving  between  $6,000  and  $10,000  in  1981  in 
improved  productivity  due  to  its  child  care  center.  Further,  49  percent  of  employers  surveyed  in 
another  study  reported  positive  effects  on  productivity. 

4.  Turnover.  An  Indiana  plastics  firm  reported  a  drop  in  turnover  from  55  percent  to  under 
10  percent  in  the  first  2  years  of  running  a  Learning  Center  and  after-school  program  for  its 
employees’  children. 


25 


5.  Absenteeism.  A  Texas  medical  device  manufacturer  claimed  it  saved  15,000  hours  from 
reduced  absenteeism  when  it  opened  a  near-by  child  care  facility.  This  reduction  combined  with 
decreased  turnover  translated  to  more  than  $2  million  in  savings  in  the  first  2  years  of  operation. 

6.  Public  Relations.  A  survey  of  Fortune  1500  firms  found  that  60  percent  of  companies 
polled  reported  an  increase  in  favorable  publicity  resulting  from  their  efforts  in  child  care  services. 
Such  coverage  increases  a  company’s  ability  to  attract  new  employees. 

7.  Morale.  Ninety  percent  of  employers  surveyed  reported  an  improvement  in  employee 
morale  as  a  result  of  their  child-care  assistance  efforts. 

Because  the  Department  of  Labor  (1988)  report  does  not  provide  complete  information  about 
the  source  of  these  findings  or  the  way  in  which  calculations  were  made,  it  is  difficult  to  determine 
their  accuracy  and  objectivity.  Some  appear  to  be  subjective,  self-reports  of  positive  program 
outcomes.  If  even  a  portion  of  these  citations  represents  the  results  of  sound  program  evaluation, 
they  indicate  the  positive  impact  that  employer-sponsored  child  care  may  have.  Further  research  is 
needed  to  support  these  claims.  In  addition,  a  measure  of  parent  satisfaction  with  employer- 
sponsored  child  care  programs  can  provide  another  perspective  on  effectiveness. 

Burud,  Aschbacher,  and  McCroskey  (1984),  in  a  well-done  “how-to”  book  on  employer- 
sponsored  child  care,  reported  on  the  results  of  the  National  Employer  Supported  Child  Care 
Project  conducted  in  1982.  This  effort  surveyed  415  firms  with  active  employer-sponsored  child 
care  programs.  Site  visits  were  made  to  some  firms  participating  in  the  study.  Firms  were  asked  to 
indicate  whether  their  child  care  programs  had  had  positive  effects  in  a  number  of  areas.  Firms 
responded  based  on  whatever  data  they  had  to  make  such  judgments;  in  many  cases,  responses 
were  entirely  subjective.  In  the  area  of  turnover,  65  percent  reported  positive  outcomes,  but  only 

1 8  companies  could  compare  turnover  rates  between  users  and  non-users  of  the  child  care  facilities. 
Of  these  18,  the  average  turnover  rate  among  all  employees  was  44  percent;  among  users,  it  was 

19  percent.  In  the  area  of  recruitment,  85  percent  of  respondents  indicated  positive  changes  had 
accompanied  program  implementation.  Only  10  firms  estimated  the  annual  recruitment  value  of 
their  child  care  program— among  these  firms  it  averaged  $16,400.  Other  positive  outcomes  reported 
were;  productivity  improvement  (reported  by  49%),  reduction  in  absenteeism  (53%),  and  decrease 
in  tardiness  (39%). 

In  other  outcome  areas,  objective  measures  of  positive  outcomes  were  not  mentioned.  For 
example,  90  percent  reported  an  improvement  in  morale,  83  percent  reported  an  improvement  in 
job  satisfaction,  73  percent  reported  an  improvement  in  commitment  to  the  organization,  and  85 
percent  reported  an  improvement  in  corporate  image.  While  several  case  studies  were  presented  in 
which  substantial  positive  improvements  were  reported  (a  60%  reduction  in  turnover  in  one  case, 
a  3%  reduction  in  absenteeism  translated  to  $89,856  savings  per  year  in  another,  and  one  with  a 
7.8%  reduction  in  turnover  translatable  to  $159,600  yearly  savings  in  recruitment  and  training 
costs)  the  methodology  used  to  determine  these  figures  was  not  described.  Without  clear 
explanation  for  the  basis  of  these  calculations  and  comparisons,  it  is  unclear  whether  they  are 
sound. 

In  addition  to  employer-sponsored  child  care  centers,  the  Department  of  Labor  (1988)  review 
also  found  that  employers  are  supporting  a  variety  of  other  assistance  programs.  These  include 
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efforts  to  sponsor  child  care  facilities  jointly  with  other  firms  in  the  area;  financial  assistance  in  the 
form  of  direct  reimbursement,  voucher  systems,  or  employer-sponsored  discounts  at  local  centers; 
flexible  benefit  programs  that  include  child  care  as  one  option;  resource  and  referral  services;  and 
innovative  work  policies  that  allow  job-sharing,  sick  child  leave,  flextime  schedules,  or  work-at- 
home  options. 

The  Department  of  Labor  (1988)  report  also  discussed  many  of  the  critical  issues  in  child  care 
including  availability,  affordability,  and  quality.  Effective  evaluation  in  this  area  would  utilize 
measures  designed  to  assess  child  care  programs  on  these  important  dimensions.  None  of  those 
cited  above  do  so. 

In  1982,  GAO  reviewed  the  status  of  military  child  care  programs.  As  with  most  GAO  studies, 
this  was  not  a  full-scale  evaluation  effort.  These  studies  tend  to  accentuate  shortcomings  and 
problems  in  the  area  of  review  and  rarely  identify  positive  findings.  This  GAO  report  provides 
information  about  the  benefits  and  shortcomings  of  military  child  care  efforts.  GAO  found  that 
many  facilities  were  neither  safe  nor  suitable  places  for  child  care,  user  fees  were  insufficient  to 
support  maintenance  and  renovation  of  facilities,  and  adequate  standards  for  care  were  lacking.  A 
more  recent  GAO  review  (1988)  found  similar  shortcomings  in  military  child  care  programs.  At 
that  time,  GAO  estimated  that  25,000  military  children  were  on  waiting  lists  for  on-base  child  care. 

Nesenholtz  (1976)  surveyed  directors  of  DoD  day  care  centers  to  obtain  information  about  size 
and  operating  characteristics,  costs,  and  staffing.  Because  no  on-site  observations  were  made  of 
day  care  centers,  all  conclusions  are  limited  by  the  self-report  nature  of  the  data.  Centers  operated 
for  a  mean  of  76  hours  per  week;  none  operated  for  less  than  48  hours  a  week.  Eighty-nine  percent 
of  children  in  these  day  care  centers  were  military  dependents.  Sixty-four  percent  were  4  years  old 
or  less.  The  mean  average  daily  attendance  was  73  full-time  children  and  72  part-time  children. 
Costs  for  day  care  varied  widely  from  site  to  site.  The  mean  fee  for  full-time  care  was  $17.59  per 
week  (ranging  from  $12  to  $25).  Day  care  center  directors  indicated  that  the  two  most  critical 
problems  they  faced  were  transience  and  their  nonappropriated  fund  status.  Being  nonappropriated 
fund  facilities,  day  care  centers  must  be  self-sustaining  or  profit-making.  Balancing  the  demand  for 
profit  against  the  costs  required  to  provide  quality  service  was  a  particular  problem. 

Resource  Consultants,  Inc.  (1980)  compared  Navy’s  child  care  capabilities  and  expenditures  to 
those  of  the  other  services.  In  comparing  the  services’  capacity  to  provide  child  care  per  10,000  in 
end  strength,  results  indicate  that  the  Navy  can  provide  child  care  to  only  1.6  children,  while  the 
Army  can  provide  care  to  almost  2.25  children,  and  the  Air  Force  to  almost  3  children.  Comparing 
expenditures  per  child  cared  for,  the  Air  Force  spent  $920  per  child  during  FY79.  The  the  Navy 
spent  only  $516  per  child.  The  authors  conclude  that  Navy’s  quality  of  care  is  lagging  the  other 
services. 

It  appears  that  little  progress  has  been  made  in  adequately  assessing  the  impact  of  employer- 
sponsored  child  care  programs  since  Miller  (1984)  did  his  review.  There  is  substantial  anecdotal 
evidence  and  some  empirical  data  to  suggest  these  programs  are  very  effective.  Until  better 
research  is  done  in  this  area,  we  cannot  assume  these  efforts  are  cost  effective.  Further,  no  attempt 
has  been  made  as  yet  to  determine  the  relative  effectiveness  of  the  many  alternative  forms  of 
employer-sponsored  child  care  support  that  exist.  Nonetheless,  if  the  Navy  is  to  remain  a 
competitive  employer,  it  must  continue  to  assist  service  members  in  obtaining  child  care. 
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Personal  Services 


Within  the  personal  services  functional  area,  research  was  found  addressing  several 
components  of  the  family  support  program.  Family  support  includes  many  unique  Navy  programs: 
Family  Service  Centers  (FSCs),  Family  Advocacy  Program,  Spouse  Employment  Assistance 
Program,  OMBUDSMAN  Program,  and  others.  Some  of  these  have  parallel  programs  either  in 
coiporate  employment  situations  (spouse  employment  assistance,  pre-retirement  counseling)  or  in 
the  public  sector  assistance  arena  (spouse  and  child  violence  programs).  Some  have  no  comparable 
program  outside  of  the  military. 

Family  Service  Centers 

Bishop,  Peters,  and  Wooley  (1982)  interviewed  191  FSC  clients,  staff  members,  and 
representatives  of  operational  commands  at  four  sites  where  FSCs  had  been  in  place  for 
approximately  1  year.  Interviews  assessed  participants’  opinions  about  the  impact  of  FSCs  on 
individuals  and  on  the  Navy’s  mission.  The  authors  concluded  that  FSCs  had  a  direct  positive 
impact  on  clients  served  and  on  commands.  This  was  a  qualitative  exploratory  study  based  on  a 
small  sample  which  bears  all  of  the  research  limitations  of  a  case-study  approach.  The  study  does 
not  provide  definitive  or  statistically  valid  measures  of  impact  in  any  specific  area  or  on  any 
particular  Navy  population. 

Although  not  a  true  evaluation  study,  Szoc  (1982)  reported  on  the  relationship  of  family  factors 
to  retention  in  the  Navy,  based  on  the  survey  responses  of  1,417  enlisted  personnel  and  officers 
whose  term  of  enlistment  or  minimum  service  requirement  was  nearing  an  end.  Analyzing  the 
simultaneous  effects  of  22  variables  and  scales  using  a  path  analysis,  the  author  found  that  opinions 
of  the  spouse  and  satisfaction  with  family  life  in  the  Navy  directly  influenced  retention  decisions. 
Retention  is  indirectly  influenced  by  the  degree  of  perceived  social  support  and  satisfaction  with 
Navy  service.  Similarly,  Bruce  and  Burch  (1989),  analyzing  survey  data  from  aviation  officers,  and 
Mohr,  Holzbach,  and  Morrison  (1981),  using  data  from  surface  warfare  officers,  found  that  spousal 
support  for  an  officer’s  career  was  strongly  related  to  the  officer’s  intent  to  pursue  a  Navy  career. 

The  effects  of  FSC  family  programs  on  satisfaction  with  military  life  and  retention  in  the 
Marine  Corps  were  estimated  by  Calvin  (1987).  Calvin  constructed  a  behavioral  model  of  Marine 
Corps  separation  using  data  from  the  1985  DoD  member  survey,  rather  than  data  from  an 
instrument  designed  for  that  purpose.  As  a  result,  the  effect  of  spousal  pressure  on  intentions  to 
leave  the  service  could  not  be  investigated.  Satisfaction  with  family  services  was  found  to  be  a 
direct  factor  in  the  decision  of  Marines  to  remain  in  the  Corps.  It  was  also  estimated  that  the  Marine 
Corps  retention  rate  would  decline  by  1  percent  if  family  services  were  to  be  eliminated.  These 
findings  were  interpreted  in  cost-benefit  terms. 

A  system  for  systematically  collecting  needs  assessment,  satisfaction,  and  utilization  data  for 
evaluating  Navy  family  support  services  on  a  recurring  basis  has  been  developed  by  Kerce  (1987a). 
The  system  is  based  on  a  stakeholder  approach  and  provides  for  data  collection  from  service 
members,  family  members,  service  providers,  career  counselors,  and  commanding  officers  and 
other  representatives  of  commands  in  the  FSC  catchment  areas.  Assessments  are  conducted  locally 
with  provisions  for  data  to  be  aggregated  at  the  program  level.  A  number  of  FSCs  have 
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implemented  this  plan  at  the  local  level.  Kerce  (1988)  reports  on  the  results  of  the  prototype 
implementation  at  Taylor-Leaver  FSC  in  San  Diego. 

Kerce  (1987b)  reports  on  an  attempt  to  utilize  existing  Navy  statistical  indicators  to  evaluate 
the  impact  of  FSCs.  The  report  contains  an  assessment  of  services  provided  by  FSCs  and 
populations  served;  a  conceptual  model  indicating  relationships  between  services  provided, 
intermediate  outcomes,  and  effects  on  the  Navy;  and  trend  analyses  of  area  cohorts  based  on  year 
of  FSC  establishment.  The  use  of  loss  data,  health  history  files,  records  of  child  and  spouse  abuse, 
desertion  data,  and  supplemental  items  on  the  enlisted  retention  questionnaire  were  investigated. 
The  results  of  this  effort  indicated  that  currently  available  Navy  data  bases  are  inadequate  to 
support  efforts  to  evaluate  FSC  impact  on  the  Navy’s  mission.  It  was  concluded  that  supplemental 
data  collection  strategies  must  be  implemented,  which  will  include  both  subjective  measures  and 
better  objective  measures  that  are  more  directly  related  to  services  provided.  The  approach  taken 
by  Kerce  (1988)  is  laudable,  in  that  existing  data  sources  were  examined  first,  before  new  data 
collection  efforts  were  proposed. 

Spouse  and  Child  Abuse/Family  Advocacy  Program 

There  are  few  available  studies  of  programs  designed  to  prevent  or  eliminate  family  abuse.  All 
of  those  found  were  public  sector  programs;  no  reports  of  abuse  programs  supported  by  private 
sector  firms  were  identified.  Of  the  reports  presented  here,  about  half  describe  military  abuse 
treatment  programs. 

Barrett  (1978)  describes  an  alternative  child  abuse  treatment  approach  that  attempts  to 
minimize  family  disruption  and  maximize  protection  of  the  child.  This  system,  Nashville’s 
Comprehensive  Emergency  Services  (CES)  system,  provides  24-hour  coordinated  services 
through  interagency  planning  and  cooperation.  The  system  provides  the  following  services:  24- 
hour  emergency  intake,  emergency  caretakers  and  homemakers,  emergency  shelters  for  families 
and  adolescents,  foster  family  homes,  and  outreach/follow-through  services.  Fully,  51  percent  of 
the  referrals  were  after  normal  working  hours,  so  the  24-hour  basis  of  this  program  was  very 
important.  Evaluation  of  Nashville’s  demonstration  project  revealed  a  56  percent  reduction  in  the 
number  of  neglect  and  dependency  (N&D)  petitions  filed,  a  51  percent  decrease  in  the  number  of 
children  placed  in  any  sort  of  substitute  care,  and  a  180  percent  increase  in  the  number  of 
complaints  in  which  no  N&D  petition  was  required. 

However,  when  Johnson  (1978)  compared  the  long-term  effectiveness  of  this  program  to  a 
more  traditional  protective  service  system  in  Savannah,  the  results  were  not  as  positive.  Neither 
system  was  effective  in  preventing  serial  abuse.  Under  Nashville’s  system,  44.8  percent  of  cases 
included  in  the  3-year  evaluation  period  had  been  reported  and  investigated  one  or  more  times  prior 
to  the  most  current  incident.  Under  Savannah’s  system,  24.4  percent  were  serial  abuse  cases, 
suggesting  that  the  traditional  approach  is  marginally  more  effective  in  preventing  serial  abuse. 
Under  both  systems,  children  involved  in  serial  abuse  situations  tended  to  be  harmed  more 
seriously  in  subsequent  incidents  than  in  the  initial  incident,  indicating  that  violence  tends  to 
escalate  over  time.  The  evaluation  revealed  that  in  both  systems  little,  if  any,  “treatment”  was 
provided  to  parents  before  children  were  returned  to  their  homes.  The  author  concludes  that 
innovation  per  se  will  not  necessarily  improve  child  protective  services.  This  evaluation  also 
highlights  the  importance  of  multiple  evaluation  criteria.  Looking  only  at  the  results  presented  by 
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Barrett,  one  would  conclude  the  new  system  was  very  effective.  However,  Johnson’s  analysis 
reveals  that  not  all  outcomes  were  positive. 

Shannon  and  St.  John  (1978)  reported  on  a  Parent  Partner  program  implemented  through  the 
Carswell  Air  Force  Base  Mental  Health  Clinic  that  was  designed  to  prevent  abuse  and  neglect.  The 
program  involves  group  meetings  among  new  parents  and  identified  or  suspected  abusive  parents 
and  one-on-one  interactions  with  volunteer  Parent  Partners.  Group  meetings  include  presentations 
and  discussions  of  topics  such  as  military  life  and  stress,  building  a  better  marriage,  sexuality,  child 
development  and  behavior,  and  child  care.  While  objective  measures  of  the  program’s 
effectiveness  were  not  reported,  participants’  evaluations  were.  Group  members  reported  that  then- 
knowledge  in  areas  covered  by  the  program  had  increased  and  that  they  had  applied  what  they  had 
learned.  They  reported  feeling  better  equipped  to  be  parents,  feeling  better  about  themselves,  and 
feeling  less  isolated. 

Neidig  (1986)  described  a  spouse  abuse  treatment  program  developed  and  implemented  in 
conjunction  with  the  Marine  Corps  FSCs  at  Perris  Island  and  Beaufort,  South  Carolina.  Based  on 
a  review  of  the  literature  on  spouse  abuse  a  new  program  was  developed.  The  Domestic  Conflict 
Control  Program  (DCCP)  was  a  program  of  10  weekly  2-hour  sessions  of  group  instruction  for 
abusive  couples.  The  approach  involves  behavioral  rehearsal,  skill-building,  and  cognitive 
restructuring  in  the  areas  of  anger  control,  stress  management,  and  communication.  Neidig 
identifies  two  procedures  used  to  assess  interspousal  violence  after  completion  of  the  training 
program.  First,  military  police  reports  of  domestic  disturbances  were  reviewed  on  a  daily  basis  to 
determine  if  participants  had  been  involved.  Second,  at  6-month  intervals,  follow-up  phone  calls 
were  made  independently  to  husbands  and  wives  concerning  additional  violent  episodes.  The 
author  admits  the  many  limitations  with  these  methods.  Both  methods  focus  only  on  those  who 
have  remained  in  the  service,  and  the  phone-call  methodology  suffers  from  lack  of  candor 
associated  with  self-report  data.  No  data  for  either  method  were  reported  by  Neidig.  As  an  alternate 
approach  to  assessing  program  effectiveness,  Neidig  compared  pre-  and  post-program  scores  of 
participants  on  locus  of  control  and  dyadic  adjustment  scales.  Both  have  been  found  to  have  utility 
in  differentiating  abusive  from  non-abusive  samples.  He  found  that  participation  in  the  program 
significantly  changed  scores  on  these  scales  in  a  direction  approximating  non-abusive  norms.  The 
need  for  adequate  long-teim  follow-up  data  was  discussed. 

McNelis  and  Await  (1986)  report  on  a  cross-service  project  to  prevent  and  treat  family  violence 
in  Hawaii.  The  Services  Assisting  Family  Environment  (SAFE)  program  provided  the  following 
services:  a  family  violence  shelter;  clinical  services,  outreach,  and  prevention  teams;  and  Wellness 
in  the  Home  (WITH)  prenatal  prevention  program  for  high-risk  mothers.  With  yearly  operating 
costs  averaging  $865,000  in  the  first  3  years  of  operation,  SAFE  had  monthly  utilization  rates  of 
330  bed  days  in  the  shelter,  215  active  cases  in  clinical  services,  and  101  active  cases  in  the  WITH 
program.  Based  on  results  not  reported  in  the  article,  the  services  institutionalized  this 
demonstration  project.  Full  evaluation  to  determine  the  effectiveness  of  the  program  in  preventing 
and  eliminating  family  violence,  the  relative  effectiveness  of  components  of  the  program,  and  the 
cost-benefits  related  to  the  SAFE  project  is  underway.  The  authors  conclude  that  the  program  is 
successful  because  it  is  being  increasingly  used  by  the  military  community  and  because  clients 
express  satisfaction  with  services  received.  This  is  positive  preliminary  evidence  of  program 
success,  but  not  enough  to  make  a  final  judgment  of  its  effectiveness. 
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A  1979  GAO  (1979b)  study  of  military  child  advocacy  programs  identified  many  problems 
with  administration  and  management  of  these  programs.  Consistent  DoD-wide  guidance  was 
lacking,  maltreatment  reporting  systems  were  incomplete  and  ineffective,  and  ;nsufficient 
command  attention  and  funding  were  provided  to  these  programs. 

Objective  evaluations  of  the  Navy’s  spouse  and  child  abuse  prevention  and  treatment  programs 
were  not  found.  To  date,  efforts  in  this  area  have  focussed  on  identifying  and  reporting  abuse  more 
than  on  treatment  programs.  Bowen,  Woolley,  and  McGaughey  (1983),  Bowen  (1983),  and  Bycer, 
Fluke,  Allen,  Schene,  and  Suski  (1984)  reviewed  and  reported  on  the  demographics  of  family 
violence  in  the  Navy  and  the  status  of  the  Navy’s  Family  Advocacy  Program  (FAP).  Although  this 
work  does  not  represent  a  full-scale  evaluation  effort,  it  does  identify  strengths  and  weakness  of 
the  program  and  point  to  the  need  for  evaluation  in  this  area.  In  fact,  Bowen,  Woolley,  and 
McGaughey,  reporting  on  site  visits  to  13  Navy  and  Marine  Corps  FAPs,  found  there  were  “no 
efforts  to  evaluate  the  effectiveness  of  the  local  FAP”  (p.  113),  despite  DoD  guidance  calling  for 
evaluation. 

Counseling  Services 

Cairo  (1983)  reviewed  the  literature  on  personal  and  career  counseling  in  business  and  industry 
and  summarized  information  found  on  the  extent  and  effectiveness  of  counseling  programs.  In 
general,  he  found  a  limited  amount  of  information  about  the  extent  of  these  programs,  particularly 
among  smaller  companies;  few  detailed  reports  describing  program  components  and  client 
characteristics;  a  lack  of  adequate  evaluation  efforts;  and  a  predominance  of  superficial 
descriptions  of  unevaluated  programs  and  well-intentioned  “how-to”  suggestions. 

In  ‘He  personal  counseling  area,  Cairo  found  that  alcohol  treatment  programs  are,  by  far,  the 
most  frequently  offered  by  industry.  These  programs  were  discussed  previously  and  will  not  be 
discussed  here.  Cairo  identified  one  study  that  attempted  to  determine  the  extent  to  which  industry 
offers  personal  counseling  to  its  employees.  The  American  Society  for  Personnel  Administrators 
(ASPA)  and  the  Bureau  of  National  Affairs  (1978)  surveyed  a  random  sample  of  ASPA  members 
and  65  responded.  One-half  of  those  responding  indicated  they  offered  in-house  counseling  of 
some  sort.  However,  it  was  often  described  as  informal,  sometimes  consisting  of  no  more  than  one- 
on-one  discussions  with  supervisors  prior  to  referral  for  outside  help.  In  fact,  a  large  majority  of 
respondents  indicated  their  programs  were  poor  or  only  fair  in  dealing  with  employees’  problems. 

Only  one  evaluation  study  was  found.  Weiner,  Akabas,  and  Sommer  (1973)  helped  to  evaluate 
a  corporate- sponsored  mental  health  care  clinic  for  clothing  industry  workers  in  New  York  City. 
Over  a  3  year  period,  718  workers  were  referred  to  the  clinic  and  442  were  provided  with  service. 
They  found  no  difference  in  a  measure  of  productivity  of  these  employees  when  compared  to  a 
matched  random  sample  of  other  industry  workers. 

Two  other  examples  of  employer- sponsored  counseling  services  were  described.  One  at  United 
States  Steel  provided  in-house  services  and  referral  for  employees.  Self-report  evaluation  data 
indicate  the  majority  of  employees  utilizing  the  services  said  the  presenting  problem  had  improved 
(80%)  and  said  they  had  received  the  type  of  treatment  they  wanted  (74%).  Another  at  Kennecott 
Copper  provided  referral  services  for  employees  to  available  community  services.  Skidmore, 
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Balsam,  and  Jones  (1974)  reported  that  of  150  employees  receiving  counseling  through  the 
program  for  absenteeism  problems,  attendance  improved  by  an  average  of  52  percent. 

In  the  area  of  career  counseling,  Cairo  (1983)  found  three  surveys  of  career  planning  practices 
in  industry  that  revealed  these  services  are  fairly  common.  However,  no  evaluations  of  the 
effectiveness  or  cost-benefit  of  these  programs  were  found.  There  is  some  peripherally-related 
literature  available  on  career  interventions,  career  assistance,  and  occupational  information  with 
population  outside  of  business  (e.g.,  with  graduating  students). 

In  the  area  of  pre-retirement  counseling,  Cairo  (1983)  found  that  efforts  in  this  area  are  fairly 
common  in  larger  firms.  However  pre-retirement  programs  vary  widely,  from  one-session 
information  meetings  with  a  benefits  administrator  to  intensive  group  counseling  programs.  Efforts 
to  evaluate  pre-retirement  counseling  programs  typically  use  self-report  outcome  measures  such  as 
attitude  toward  retirement,  knowledge  of  retirement  issues,  acquisition  of  relevant  retirement 
information,  and  pre-retirement  activities,  and  rarely  use  any  control  or  comparison  group.  Cairo 
concludes  that  the  information  available  is  insufficient  to  determine  the  effectiveness  of  these 
programs. 

Miller  (1981)  reviewed  Polaroid’s  Employees’  Counseling  Service  begun  in  1958.  The  service 
treats  workers  with  problems  related  to  work  (e.g.,  conflict  with  supervisor,  anxiety  about  job 
performance),  problems  related  to  self  (e.g.,  depression,  emotional  disturbances,  alcoholism),  and 
problems  related  to  others  (e.g.,  marital  problems,  parent-child  problems,  premarital  concerns). 
Polaroid  believes  that  employees  who  are  “anxious,  upset,  or  lack  self  confidence  are  seriously 
blocked  in  their  ability  to  perform”  (p.  50).  Several  other  references  to  the  impact  of  personal 
problems  on  job  performance  are  made  by  Miller,  but  no  empirical  evidence  is  provided  to  support 
this  claim.  Nor  does  he  provide  evidence  that  participating  in  treatment  results  in  improvements  in 
job  performance.  Rather,  in  reference  to  measuring  productivity  improvements,  he  says  “It  is 
difficult  to  measure,  but  a  conservative  estimate  of  the  productivity  gain  of  those  who  have  been 
counseled  is  10  percent”  (p.  55).  Projecting  this  estimate  to  the  entire  population  seen  by  the 
counseling  department,  Miller  states  that  every  $1.00  spent  for  counseling  yields  a  $5.00  payback. 
This  subjective  cost-benefit  analysis  is  not  enough  to  allow  confident  decisions  to  be  made  about 
such  programs. 

Esteban  (1985)  discusses  the  growing  need  for  retirement  planning  programs  and  the 
components  of  an  effective  program.  He  states  that  retirement  planning  programs  can  “provide  a 
demonstration  of  corporate  concern  for  older  workers,  provide  visible  proof  of  its  social 
conscience,  maintain  an  open  environment  in  the  workplace,  and  boost  morale  and  enhance 
productivity”  (p.  22).  No  empirical  data  are  provided  to  support  this  claim. 

The  staff  of  Catalyst  report  (1986)  that  in  198 1  just  4  percent  of  companies  responding  to  one 
survey  indicated  they  provided  spouse  employment  assistance.  In  another  survey  conducted  in 
1985,  28  percent  indicated  they  offer  such  services.  With  the  increasing  number  of  two-career 
couples,  these  services  are  becoming  more  important.  Catalyst  repons  that  the  potential  benefits  of 
spouse  employment  assistance  programs  include  increasing  relocation  success,  decreasing 
reluctance  to  move,  offering  an  incentive  to  recruits,  reducing  the  stress  and  expense  of  relocation, 
and  indicating  corporate  concern  for  employees’  personal  and  family  needs.  No  empirical  evidence 
was  provided  to  indicate  such  outcomes  have  ever  been  investigated. 
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Commissaries/Exchanges 


Resource  Consultants,  Inc.  (1980)  in  comparing  the  combined  Army/Air  Force  exchange 
program  to  that  of  the  Navy,  found  that  the  Navy  spends  slightly  more  per  capita  on  its  exchanges 
than  do  the  other  services.  However,  per  capita  sales  are  fairly  comparable  between  the  Navy  and 
Army/Air  Force  systems. 

Religious  Support 

Hawkins  (1980)  conducted  a  preliminary  evaluation  of  a  counter-attrition  program 
implemented  within  the  religious  support  functional  area.  Chaplains  Religious  Enrichment  and 
Development  Operation  (CREDO)  is  a  spiritually-based  program  first  established  in  1971  to  assist 
marginal  performers,  naval  personnel  with  various  problems,  and  disciplinary  offenders.  Hawkins 
mentions  an  unreferenced  survey  evaluation  of  CREDO  that  indicated  respondents  believed 
CREDO  reduced  the  likelihood  of  premature  attrition  from  the  Navy.  No  additional  description 
was  provided  of  the  study  methodology  or  results. 

The  purpose  of  Hawkins’  effort  was  to  evaluate  an  experimental  implementation  of  CREDO  at 
Recruit  Training  Center  (RTC),  Great  Lakes.  This  program,  known  as  RTC  CREDO,  was 
implemented  in  May  1979  and  continued  throughout  March  1980.  His  evaluation  focussed  on  the 
impact  of  RTC  CREDO  on  first-term  enlisted  attrition.  Because  the  evaluation  was  done  when 
participants  had  been  in  the  service  for  a  brief  period  of  time  (1  to  9  months),  it  represents  only  a 
tentative  evaluation  of  the  program’s  effectiveness.  Subjects  were  male  recruits  from  the 
Apprentice  Training  Division  (General  Detail  personnel).  Some  were  selected  to  participate  in 
RTC  CREDO  based  on  review  of  their  performance  and  discipline  records;  others  were  randomly 
selected.  Comparison  group  personnel  were  retrospectively  matched  to  recruits  assigned  to  RTC 
CREDO  based  on  record  review  or  were  those  not  randomly  selected  for  participation.  In  essence, 
Hawkins  used  two  experimental  groups  and  two  control  groups.  He  cites  some  problems  in  the 
comparability  of  these  groups,  however.  Us;ng  regression  analyses,  he  found  that  being  assigned 
to  either  experimental  group  was  the  best  predictor  of  non-attrition.  Based  on  this  preliminary 
evaluation,  it  would  appear  that  CREDO  programs  may  hold  promise  and  should  be  evaluated 
more  rigorously. 

Multi-program  Evaluation 

A  pamphlet  recently  published  by  the  Department  of  the  Army  (1989)  describes  its  overall 
Quality  of  Life  program  and  provides  minimum  standards  to  be  used  in  evaluating  quality  of  life 
for  the  Army  community.  Standards  on  nine  dimensions  are  provided  against  which  the 
performance  of  various  programs  may  be  evaluated.  These  dimensions  are  related  to  the  manner  in 
which  services  are  provided.  They  specify  such  factors  as  accessibility,  availability,  timeliness, 
space  requirements,  facilitie  s,  and  the  population  which  is  eligible  to  receive  each  of  the  services. 
The  specification  of  implementation  standards  should  be  an  integral  part  of  an  overall  evaluation 
because  goal  attainment,  attitudinal  enhancements,  and  final  outcomes  may  all  be  affected  by  the 
implementation  process. 
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CONCLUSIONS 


Based  on  the  review  of  the  literature  presented  above,  several  conclusions  can  be  drawn. 
Except  in  certain  areas,  there  is  a  pervasive  lack  of  research  designed  to  evaluate  quality  of  life 
programs.  At  this  time,  we  do  not  know  the  effectiveness  of  many  of  these  programs  in  meeting 
their  goals,  their  effects  on  other  factors  such  as  retention,  performance,  and  readiness,  nor  their 
cost-effectiveness.  Many  of  the  evaluation  efforts  found  suffer  from  a  lack  of  control  and  the  sole 
reliance  on  subjective  data.  Subjective  evaluation  is  very  important  in  the  quality  of  life  field,  but 
should  not  be  used  at  the  exclusion  of  objective  measures  when  they  are  available  or  can  be  readily 
developed  or  obtained.  Finally,  because  evaluations  become  dated  very  quickly  as  programs, 
policies,  and  the  environment  change,  the  validity  of  previous  evaluation  results  for  current  quality 
of  life  programs  is  unknown. 

In  three  areas,  research  was  found  that,  to  an  extent,  contradicts  the  above  conclusions.  Some, 
but  certainly  not  all,  of  the  studies  evaluating  alcohol  and  drug  control  efforts,  health  promotion 
programs,  and  employer-sponsored  child  care  programs  are  more  rigorous  and  objective  than  those 
in  other  areas.  The  effective  evaluations  use  comparison  or  control  groups  and/or  pre-post 
comparisons  to  evaluate  effects  of  these  programs.  Further,  there  are  more  reports  of  measured 
outcomes  linked  to  these  programs  and  many  of  these  evaluations  assess  the  effects  of  such 
programs  on  more  than  one  outcome,  including  client  satisfaction.  In  this  area,  there  were  a  number 
of  studies  that  conducted  cost-effectiveness  analyses.  The  methodologies  used  in  some  of  the  better 
evaluations  in  these  areas  can  be  used  as  the  basis  for  developing  more  effective  evaluations  in 
other  areas  as  well. 

In  some  quality  of  life  arenas,  there  are  major  problems  evaluating  program  outcomes  because 
outcome  areas  are  not  clearly  defined  and  because  process  measures  are  not  collected.  In  some 
areas,  service  member  satisfaction  with  services  provided  or  overall  satisfaction  with  quality  of  life 
may  be  the  only  feasible  measure  available.  In  most  cases,  client  satisfaction  should  certainly  be 
one  outcome  that  is  measured.  The  problems  of  separating  out  the  effects  of  one  program  (e.g., 
military  pay)  on  a  given  outcome  measure  (e.g.,  retention)  from  the  effects  that  all  other 
concomitant  factors  in  the  environment  have  on  the  same  outcome  measure  (e.g.,  labor  market 
situation,  other  forms  of  compensation,  training  and  education  received,  etc.)  is  difficult  in  real- 
world  research.  Particularly  within  the  compensation  area,  researchers  have  resorted  to  the  use  of 
modeling  techniques  to  assess  such  effects. 

The  studies  reviewed  here  do  not  provide  a  definitive  view  of  the  quality  of  life  programs. 
Rather,  they  can  suggest  evaluation  methodologies,  outcome  measures,  and  cost-effectiveness 
approaches  to  be  used  in  future  evaluations.  An  effective  approach  to  subsequent  evaluations  will 
include  service  member  attitude  and  satisfaction  assessment,  as  a  minimum;  objective 
effectiveness  data,  whenever  obtainable;  and  cost-effectiveness  analysis,  when  feasible. 

RECOMMENDATION 

It  is  recommended  that  efforts  to  evaluate  Navy  programs  should  include  attitude  and 
satisfaction  assessment  and  objective  effectiveness  data  as  a  minimum,  with  cost-effective  uess 
analysis  when  feasible. 
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